Oct 30 16:20:01.828239 ip-10-0-136-74 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 16:20:01.828252 ip-10-0-136-74 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 16:20:01.828276 ip-10-0-136-74 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 16:20:01.828517 ip-10-0-136-74 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 16:20:12.077892 ip-10-0-136-74 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 16:20:12.077912 ip-10-0-136-74 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3119779aa1d74a50be80ded1a93977fb --
Oct 30 16:22:09.503523 ip-10-0-136-74 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 16:22:09.866074 ip-10-0-136-74 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:22:09.866074 ip-10-0-136-74 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 16:22:09.866074 ip-10-0-136-74 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:22:09.866074 ip-10-0-136-74 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870475    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870562    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870566    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870569    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870572    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870575    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870578    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870581    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:09.870570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870583    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870586    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870589    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870592    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870595    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870598    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870600    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870603    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870605    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870608    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870610    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870613    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870615    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870618    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870621    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870623    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870626    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870629    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870631    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870634    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:09.870882 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870636    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870641    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870646    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870649    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870652    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870655    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870658    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870660    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870663    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870665    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870668    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870671    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870673    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870676    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870680    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870683    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870686    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870688    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870691    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870694    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:09.871360 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870697    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870700    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870702    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870705    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870708    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870710    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870713    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870715    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870718    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870721    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870723    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870725    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870728    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870731    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870733    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870738    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870742    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870746    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870749    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:09.871826 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870751    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870754    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870757    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870759    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870762    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870765    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870768    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870771    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870775    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870777    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870780    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870783    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870786    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870788    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870791    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870794    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870797    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870800    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870803    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870805    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:09.872287 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870808    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870810    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870813    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870815    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870818    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870820    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870824    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870827    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870830    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870832    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870835    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870837    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870841    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870844    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870846    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870849    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870852    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870855    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.870858    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:09.872759 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871685    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871693    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871696    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871699    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871701    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871705    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871707    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871710    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871713    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871716    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871719    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871722    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871725    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871728    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871731    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871733    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871736    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871738    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871741    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871744    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:09.873248 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871747    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871750    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871753    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871756    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871758    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871761    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871763    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871766    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871769    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871771    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871774    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871777    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871780    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871782    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871785    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871788    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871791    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871793    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871796    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871799    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:09.873732 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871801    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871804    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871806    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871811    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871814    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871816    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871819    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871822    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871824    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871827    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871829    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871832    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871835    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871837    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871840    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871843    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871845    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871848    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871850    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871853    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:09.874235 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871856    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871858    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871861    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871864    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871867    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871869    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871872    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871875    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871877    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871879    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871884    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871889    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871893    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871897    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871900    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871904    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871907    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871910    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871912    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:09.874710 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871915    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871918    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871921    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871923    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871926    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871928    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871931    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871934    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871936    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871939    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871941    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871944    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871946    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871949    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871952    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871954    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871959    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871962    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871965    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871967    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:09.875147 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871970    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871973    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871975    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871978    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871981    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871984    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.871987    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872553    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872560    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872565    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872568    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872571    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872574    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872577    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872580    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872583    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872585    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872588    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872590    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872593    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:09.875633 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872596    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872598    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872601    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872604    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872607    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872609    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872612    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872615    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872619    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872622    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872625    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872628    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872632    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872637    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872640    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872643    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872646    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872648    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872651    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872654    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:09.876116 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872657    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872660    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872663    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872666    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872669    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872671    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872674    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872676    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872679    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872682    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872684    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872687    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872691    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872694    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872697    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872700    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872702    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872705    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872707    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:09.876595 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872710    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872714    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872716    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872719    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872721    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872724    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872727    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872729    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872732    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872734    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872737    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872739    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872742    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872746    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872748    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872751    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872754    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872757    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872760    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:09.877041 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872762    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872765    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872768    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872770    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872773    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872776    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872778    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872781    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872783    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872786    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872788    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872791    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872793    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872796    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872800    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872803    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872805    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872808    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872810    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872813    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:09.877504 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872816    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872818    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872821    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872823    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872826    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872828    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872832    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872835    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872837    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872840    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872843    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872846    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872850    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872852    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.872855    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873848    2570 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873860    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873873    2570 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873883    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873888    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873891    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 16:22:09.878019 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873905    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873909    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873913    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873916    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873919    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873924    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873928    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873931    2570 flags.go:64] FLAG: --cgroup-root=""
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873934    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873937    2570 flags.go:64] FLAG: --client-ca-file=""
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873940    2570 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873943    2570 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873947    2570 flags.go:64] FLAG: --cluster-domain=""
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873950    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873954    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873957    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873960    2570 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873965    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873968    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873971    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873974    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873977    2570 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873981    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873985    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 16:22:09.878525 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873988    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873991    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873996    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.873999    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874001    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874004    2570 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874007    2570 flags.go:64] FLAG: --enable-server="true"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874010    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874015    2570 flags.go:64] FLAG: --event-burst="100"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874018    2570 flags.go:64] FLAG: --event-qps="50"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874021    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874024    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874027    2570 flags.go:64] FLAG: --eviction-hard=""
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874032    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874035    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874038    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874042    2570 flags.go:64] FLAG: --eviction-soft=""
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874045    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874048    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874051    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874054    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874058    2570 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874061    2570 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874063    2570 flags.go:64] FLAG: --feature-gates=""
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874068    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 16:22:09.879108 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874071    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874075    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874078    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874081    2570 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874084    2570 flags.go:64] FLAG: --help="false"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874087    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874091    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874095    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874098    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874101    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874104    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874108    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874111    2570 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874113    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874116    2570 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874119    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874122    2570 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874125    2570 flags.go:64] FLAG: --kube-reserved=""
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874128    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874131    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874135    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874138    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874141    2570 flags.go:64] FLAG: --lock-file=""
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874144    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 16:22:09.879709 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874147    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874150    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874156    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874159    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874162    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874165    2570 flags.go:64] FLAG: --logging-format="text"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874168    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874171    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874174    2570 flags.go:64] FLAG: --manifest-url=""
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874178    2570 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874195    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874199    2570 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874203    2570 flags.go:64] FLAG: --max-pods="110"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874206    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874209    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874213    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874216    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874219    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874222    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874225    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874233    2570 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874240    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874244    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874247    2570 flags.go:64] FLAG: --pod-cidr=""
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874250    2570 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 16:22:09.880354 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874253    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874256    2570 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874259    2570 flags.go:64] FLAG: --port="10250"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874262    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874266    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0886c9cb95e15c7f3"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874269    2570 flags.go:64] FLAG: --qos-reserved=""
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874272    2570 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874275    2570 flags.go:64] FLAG: --register-node="true"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874278    2570 flags.go:64] FLAG: --register-with-taints=""
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874282    2570 flags.go:64] FLAG: --registry-burst="10"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874285    2570 flags.go:64] FLAG: --registry-qps="5"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874288    2570 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874291    2570 flags.go:64] FLAG: --reserved-memory=""
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874295    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874298    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874302    2570 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874304    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874307    2570 flags.go:64] FLAG: --runonce="false"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874311    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874314    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874317    2570 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874320    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874323    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874327    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874331    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874334    2570 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 16:22:09.880944 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874337    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874340    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874343    2570 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874346    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874349    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874352    2570 flags.go:64] FLAG: --system-cgroups=""
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874355    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874361    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874367    2570 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874369    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874374    2570 flags.go:64] FLAG: --tls-min-version=""
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874377    2570 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874380    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874383    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874386    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874389    2570 flags.go:64] FLAG: --v="2"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874394    2570 flags.go:64] FLAG: --version="false"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874398    2570 flags.go:64] FLAG: --vmodule=""
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874402    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874406    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874551    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874554    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874557    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874560    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:09.881603 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874564    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874567    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874570    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874572    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874575    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874577    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874581    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874583    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874586    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874589    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874591    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874594    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874597    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874600    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874602    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874605    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874611    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874614    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874617    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874620    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:09.882176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874622    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874625    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874627    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874630    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874633    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874635    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874638    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874640    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874643    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874646    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874648    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874651    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874653    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874656    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874658    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874661    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874664    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874668    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874671    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:09.882663 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874675    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874678    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874681    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874683    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874686    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874688    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874691    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874693    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874696    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874701    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874704    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874706    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874710    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874714    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874717    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874720    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874722    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874725    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874728    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:09.883140 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874730    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874733    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874736    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874738    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874741    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874744    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874746    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874749    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874751    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874754    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874756    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874759    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874761    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874764    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874767    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874770    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874773    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874776    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874778    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874781    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:09.883609 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874783    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874786    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874791    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874793    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874796    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874799    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874801    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874804    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874806    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874809    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874811    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874815    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874817    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874820    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874823    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874825    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874828    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874831    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874833    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874836    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:09.884075 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874838    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874841    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874844    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874846    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.874873    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874911    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874916    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874920    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874923    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874926    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874928    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874931    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874934    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874937    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874942    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874946    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:09.884570 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874949    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874952    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874954    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874957    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874959    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874962    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874964    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874967    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874970    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874972    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874975    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874977    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874980    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874983    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874985    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874988    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874991    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874993    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874996    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.874999    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:09.884939 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875002    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875004    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875007    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875009    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875012    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875015    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875018    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875021    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875023    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875026    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875030    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875033    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875036    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875039    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875041    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875044    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875047    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875049    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875052    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875055    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:09.885427 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875057    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875060    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875063    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875065    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875068    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875071    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875073    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875076    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875079    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875081    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875084    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875087    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875090    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875092    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875095    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875097    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875100    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875103    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875106    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875108    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:09.885884 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875111    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875113    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875117    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875121    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875124    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875126    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875129    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875132    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875135    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875137    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875140    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875142    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875145    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875148    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875151    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875154    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875156    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875159    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875162    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:09.886424 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875164    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875167    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875170    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875172    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875174    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875177    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875179    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875195    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875199    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875202    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875207    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875211    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875215    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875217    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875221    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.875225    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.882176    2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.882210    2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.882906    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 16:22:09.886929 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.882913    2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.882973    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.882979    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.882983    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.882987    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.882990    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.882993    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.882995    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.882998    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883001    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883004    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883007    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883010    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883012    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883015    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883018    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883020    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883023    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883026    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883028    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:09.887403 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883031    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883034    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883037    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883040    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883042    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883045    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883048    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883050    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883053    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883056    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883064    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883068    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883072    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883076    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883080    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883083    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883086    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883088    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883091    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:09.887871 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883093    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883096    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883099    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883101    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883104    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883106    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883109    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883112    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883116    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883119    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883122    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883125    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883127    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883130    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883133    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883136    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883139    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883142    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883144    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:09.888335 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883147    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883150    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883153    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883155    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883158    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883161    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883164    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883166    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883169    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883172    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883174    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883177    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883179    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883182    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883196    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883199    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883202    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883204    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883207    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:09.888796 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883210    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883212    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883215    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883218    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883220    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883223    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883225    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883229    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883231    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883234    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883237    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883239    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883242    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883245    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883248    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883250    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883253    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883255    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883258    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883261    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:09.889419 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883264    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883266    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883269    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883271    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883273    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883276    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883278    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883281    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883284    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883286    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.883316    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883485    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883490    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883493    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883497    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883500    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:09.889908 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883503    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883506    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883508    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883511    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883514    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883517    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883521    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883525    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883528    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883531    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883534    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883537    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883539    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883542    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883544    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883547    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883550    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883553    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883555    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883558    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:09.890331 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883560    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883563    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883565    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883568    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883570    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883573    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883576    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883578    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883581    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883584    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883586    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883589    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883591    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883594    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883597    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883599    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883602    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883605    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883607    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:09.890821 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883610    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883613    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883615    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883618    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883621    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883624    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883626    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883629    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883631    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883634    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883639    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883642    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883645    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883647    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883650    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883652    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883655    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883657    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883660    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883663    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:09.891440 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883666    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883668    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883671    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883674    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883676    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883679    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883681    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883684    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883686    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883689    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883691    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883694    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883697    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883700    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883702    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883705    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883707    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883710    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883712    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883715    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:09.892225 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883717    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883720    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883722    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883725    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883728    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883731    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883733    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883736    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883738    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883741    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883743    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883746    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883749    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883751    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883754    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883757    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883759    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883762    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883765    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:09.892830 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883767    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883770    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883772    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.883794    2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883837    2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883842    2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883845    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883848    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883850    2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883853    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883856    2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883858    2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883861    2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883863    2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883866    2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883868    2570 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:09.893289 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883871    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883874    2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883877    2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883879    2570 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883882    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883885    2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883887    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883889    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883892    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883895    2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883897    2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883900    2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883902    2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883905    2570 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883907    2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883910    2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883913    2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883915    2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883918    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883920    2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:09.893743 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883923    2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883925    2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883928    2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883930    2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883933    2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883936    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883938    2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883941    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883943    2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883946    2570 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883948    2570 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883951    2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883953    2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883956    2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883959    2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883961    2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883964    2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883966    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883969    2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883971    2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:09.894285 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883974    2570 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883976    2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883979    2570 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883981    2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883984    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883986    2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883989    2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883992    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883995    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.883997    2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884000    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884002    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884005    2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884008    2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884010    2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884013    2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884016    2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884019    2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884023    2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:09.894768 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884026    2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884031    2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884034    2570 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884036    2570 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884039    2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884041    2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884045    2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884049    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884053    2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884056    2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884058    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884061    2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884063    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884066    2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884069    2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884071    2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884074    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884076    2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884079    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884082    2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:09.895257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884085    2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884087    2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884090    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884093    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884095    2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884098    2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884101    2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884103    2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884106    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884109    2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884111    2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884114    2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884116    2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884120    2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:09.884123    2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.884181    2570 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.886085    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.887039    2570 server.go:1014] "Starting client certificate rotation"
Oct 30 16:22:09.895747 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.887133    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 16:22:09.896166 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.887170    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 16:22:09.896166 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.888852    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:22:09.896166 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.894514    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:22:09.906024 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.906003    2570 log.go:25] "Validated CRI v1 runtime API"
Oct 30 16:22:09.909100 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.909083    2570 log.go:25] "Validated CRI v1 image API"
Oct 30 16:22:09.909979 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.909949    2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 16:22:09.914094 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.914075    2570 fs.go:135] Filesystem UUIDs: map[0e86a780-c38a-4fe9-9c56-21c7506d9f2a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8de81c42-5d50-4c38-a09e-cd4bbdec3d79:/dev/nvme0n1p4]
Oct 30 16:22:09.914149 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.914094    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 16:22:09.920507 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.920381    2570 manager.go:217] Machine: {Timestamp:2026-10-30 16:22:09.919017168 +0000 UTC m=+0.305896066 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103490 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2095f3ba0ace843d346f74763c0694 SystemUUID:ec2095f3-ba0a-ce84-3d34-6f74763c0694 BootID:3119779a-a1d7-4a50-be80-ded1a93977fb Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:eb:1e:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:eb:1e:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:8c:f9:d2:ee:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 16:22:09.920507 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.920500    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 16:22:09.920627 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.920619    2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 16:22:09.923023 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.922994    2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 16:22:09.923161 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.923026    2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 16:22:09.923535 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.923525    2570 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 16:22:09.923570 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.923537    2570 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 16:22:09.923570 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.923565    2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 16:22:09.923619 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.923581    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 16:22:09.924306 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.924296    2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 16:22:09.927039 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.927027    2570 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 16:22:09.927163 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.927154    2570 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 16:22:09.927907 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.927897    2570 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 16:22:09.927946 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.927911    2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 16:22:09.927946 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.927923    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 16:22:09.927946 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.927932    2570 kubelet.go:405] "Adding apiserver pod source"
Oct 30 16:22:09.927946 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.927940    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 16:22:09.928275 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.928263    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 16:22:09.930603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.930587    2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 16:22:09.935949 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.935932    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 16:22:09.936793 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.936771    2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 16:22:09.936864 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.936838    2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 16:22:09.937343 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937331    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 16:22:09.937373 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937348    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 16:22:09.937373 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937355    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 16:22:09.937373 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937361    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 16:22:09.937373 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937369    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 16:22:09.937475 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937379    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 16:22:09.937908 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937900    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 16:22:09.937941 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937910    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 16:22:09.937941 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937917    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 16:22:09.937941 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937923    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 16:22:09.937941 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937931    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 16:22:09.937941 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.937940    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 16:22:09.938948 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.938938    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 16:22:09.938980 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.938952    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 16:22:09.942729 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.942711    2570 server.go:1263] "Started kubelet"
Oct 30 16:22:09.942820 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.942778    2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 16:22:09.942889 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.942804    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 16:22:09.942889 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.942883    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 16:22:09.943484 ip-10-0-136-74 systemd[1]: Started Kubernetes Kubelet.
Oct 30 16:22:09.943850 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.943833    2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 16:22:09.945644 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.945631    2570 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 16:22:09.948725 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:09.948704    2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 16:22:09.949031 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.949016    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 16:22:09.949505 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.949484    2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 16:22:09.949779 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:09.949612    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:09.950246 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950121    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 16:22:09.950246 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950130    2570 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 16:22:09.950246 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950159    2570 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 16:22:09.950413 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950265    2570 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 16:22:09.950413 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950287    2570 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 16:22:09.950500 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950416    2570 factory.go:55] Registering systemd factory
Oct 30 16:22:09.950500 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950429    2570 factory.go:223] Registration of the systemd container factory successfully
Oct 30 16:22:09.950657 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950643    2570 factory.go:153] Registering CRI-O factory
Oct 30 16:22:09.950657 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950655    2570 factory.go:223] Registration of the crio container factory successfully
Oct 30 16:22:09.950760 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950700    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 16:22:09.950760 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950721    2570 factory.go:103] Registering Raw factory
Oct 30 16:22:09.950760 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.950734    2570 manager.go:1199] Started watching for new ooms in manager
Oct 30 16:22:09.951543 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.951527    2570 manager.go:319] Starting recovery of all containers
Oct 30 16:22:09.961603 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.961583    2570 manager.go:324] Recovery completed
Oct 30 16:22:09.968939 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.968916    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:09.971297 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.971276    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:09.971387 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.971307    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:09.971387 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.971318    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:09.972602 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.972588    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:09.972678 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.972588    2570 cpu_manager.go:226] "Starting" policy="none"
Oct 30 16:22:09.972678 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.972641    2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 16:22:09.972776 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.972679    2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 16:22:09.973512 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.973495    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:09.973566 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.973526    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:09.973566 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.973539    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:09.973818 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.973806    2570 policy_none.go:50] "Start"
Oct 30 16:22:09.973847 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.973823    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 16:22:09.973847 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.973833    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 16:22:09.974890 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.974877    2570 policy_none.go:44] "Start"
Oct 30 16:22:09.980331 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:09.980310    2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 16:22:09.981673 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.980535    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 16:22:09.981673 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:09.980520    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-74.ec2.internal.18e35b2e0b78cb8c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-74.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-74.ec2.internal,},FirstTimestamp:2026-10-30 16:22:09.942678412 +0000 UTC m=+0.329557305,LastTimestamp:2026-10-30 16:22:09.942678412 +0000 UTC m=+0.329557305,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-74.ec2.internal,}"
Oct 30 16:22:09.989274 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.989254    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 16:22:09.989369 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.989350    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-prksd"
Oct 30 16:22:09.998766 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.998737    2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 16:22:09.999569 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:09.999546    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-prksd"
Oct 30 16:22:09.999853 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:09.999771    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-74.ec2.internal.18e35b2e0d2d72de  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-74.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-74.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-74.ec2.internal,},FirstTimestamp:2026-10-30 16:22:09.971294942 +0000 UTC m=+0.358173834,LastTimestamp:2026-10-30 16:22:09.971294942 +0000 UTC m=+0.358173834,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-74.ec2.internal,}"
Oct 30 16:22:10.000411 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.000373    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 16:22:10.000411 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.000372    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 16:22:10.000411 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.000374    2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 16:22:10.001510 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.001498    2570 manager.go:341] "Starting Device Plugin manager"
Oct 30 16:22:10.001597 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.001531    2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 16:22:10.001597 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.001542    2570 server.go:87] "Starting device plugin registration server"
Oct 30 16:22:10.001800 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.001781    2570 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 16:22:10.001881 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.001795    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 16:22:10.001946 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.001883    2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 16:22:10.001995 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.001958    2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 16:22:10.001995 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.001966    2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 16:22:10.002521 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.002499    2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 16:22:10.002634 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.002546    2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.062336 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.062302    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 16:22:10.063535 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.063519    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 16:22:10.063624 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.063544    2570 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 16:22:10.063624 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.063569    2570 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 16:22:10.063624 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.063609    2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 16:22:10.076055 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.076036    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:10.102020 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.101997    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:10.103165 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.103147    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:10.103268 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.103180    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:10.103268 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.103205    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:10.103268 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.103238    2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.118316 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.118229    2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.118316 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.118262    2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-74.ec2.internal\": node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.158030 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.158004    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.164075 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.164046    2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"]
Oct 30 16:22:10.164145 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.164115    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:10.165113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.165094    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:10.165248 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.165127    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:10.165248 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.165141    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:10.166400 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.166385    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:10.166550 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.166535    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.166602 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.166568    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:10.167138 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.167114    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:10.167138 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.167137    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:10.167284 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.167138    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:10.167284 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.167147    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:10.167284 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.167161    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:10.167284 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.167174    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:10.168215 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.168198    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.168280 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.168228    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:10.169016 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.168998    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:10.169082 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.169030    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:10.169082 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.169047    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:10.202097 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.202071    2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-74.ec2.internal\" not found" node="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.207384 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.207366    2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-74.ec2.internal\" not found" node="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.251663 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.251636    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041057424ec306b79ef7363e62fcbac0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-74.ec2.internal\" (UID: \"041057424ec306b79ef7363e62fcbac0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.251663 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.251663    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.251812 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.251681    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.258754 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.258735    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.352692 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.352662    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041057424ec306b79ef7363e62fcbac0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-74.ec2.internal\" (UID: \"041057424ec306b79ef7363e62fcbac0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.352774 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.352697    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.352774 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.352716    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.352838 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.352775    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.352838 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.352778    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.352838 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.352778    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041057424ec306b79ef7363e62fcbac0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-74.ec2.internal\" (UID: \"041057424ec306b79ef7363e62fcbac0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.359777 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.359754    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.460381 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.460313    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.505511 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.505487    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.510410 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.510389    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:10.561221 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.561042    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.661563 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.661533    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.761756 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.761687    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.862625 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.862597    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:10.887884 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.887855    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 16:22:10.888378 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:10.888027    2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 16:22:10.963168 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:10.963133    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:11.002950 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.002910    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:17:10 +0000 UTC" deadline="2028-06-08 16:34:10.542692124 +0000 UTC"
Oct 30 16:22:11.002950 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.002948    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14088h11m59.539748082s"
Oct 30 16:22:11.016319 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.016248    2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:11.017757 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.017738    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:11.017872 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.017769    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:11.017872 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.017782    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:11.063357 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:11.063326    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:11.114176 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:11.114135    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6861b6533e84603bf25f2e81fd4e489.slice/crio-7ab6d0bd18409d2c903badb6074cca41ba8bcf67965b2f5cdca4078b5ec9ad34 WatchSource:0}: Error finding container 7ab6d0bd18409d2c903badb6074cca41ba8bcf67965b2f5cdca4078b5ec9ad34: Status 404 returned error can't find the container with id 7ab6d0bd18409d2c903badb6074cca41ba8bcf67965b2f5cdca4078b5ec9ad34
Oct 30 16:22:11.122832 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.122812    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:22:11.132741 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:11.132719    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041057424ec306b79ef7363e62fcbac0.slice/crio-ec42e889039300d4ce8157ac0177cafa75e1397bb94c07394e66beaedbc228dd WatchSource:0}: Error finding container ec42e889039300d4ce8157ac0177cafa75e1397bb94c07394e66beaedbc228dd: Status 404 returned error can't find the container with id ec42e889039300d4ce8157ac0177cafa75e1397bb94c07394e66beaedbc228dd
Oct 30 16:22:11.154030 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.154007    2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:11.164371 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:11.164350    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:11.261680 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.261648    2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:11.264854 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:11.264836    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:11.365967 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:11.365859    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:11.466090 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:11.466051    2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:11.558931 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.558901    2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:11.650472 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.650377    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:11.675458 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.675303    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:22:11.676662 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.676369    2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:11.700857 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.700820    2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:22:11.921820 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.921725    2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:11.929057 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.929028    2570 apiserver.go:52] "Watching apiserver"
Oct 30 16:22:11.950402 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.950362    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 16:22:11.958134 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.958104    2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 16:22:11.958533 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.958507    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-pz9z8","openshift-dns/node-resolver-2lxwn","openshift-network-diagnostics/network-check-target-htgvf","openshift-network-operator/iptables-alerter-jwpjd","openshift-ovn-kubernetes/ovnkube-node-66645","openshift-image-registry/node-ca-rc86k","openshift-multus/multus-additional-cni-plugins-jv959","openshift-multus/multus-k545t","openshift-multus/network-metrics-daemon-pv5p7","kube-system/konnectivity-agent-j6fh2","kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal","kube-system/global-pull-secret-syncer-x4pp5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"]
Oct 30 16:22:11.961330 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.961309    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:11.963923 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.963901    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k545t"
Oct 30 16:22:11.966381 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.966360    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:11.966486 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:11.966444    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:11.968857 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.968834    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:11.971712 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.971689    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:11.974522 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.974502    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:11.976748 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.976730    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:11.976874 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:11.976805    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:11.977305 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.977283    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 16:22:11.977458 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.977325    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 16:22:11.977458 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.977372    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 16:22:11.977458 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.977386    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 16:22:11.977458 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.977283    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 16:22:11.977754 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.977739    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6852\""
Oct 30 16:22:11.978750 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.978659    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-86hj4\""
Oct 30 16:22:11.978841 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.978753    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 16:22:11.979350 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.979326    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:11.981396 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.981376    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:11.984092 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.983973    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:11.987026 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.987007    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:11.989107 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:11.989081    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:11.989213 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:11.989157    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:12.001711 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.001687    2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 16:22:12.013390 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.013361    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cgh8n"
Oct 30 16:22:12.026444 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.026388    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cgh8n"
Oct 30 16:22:12.052148 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.052114    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 16:22:12.061521 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.061700 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061534    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-dbus\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.061700 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061564    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-env-overrides\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.061700 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061621    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:12.061700 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061660    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a046e81-b1ab-4da2-ac50-7d06f2c30668-hosts-file\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.061932 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061708    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.061932 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061734    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-node-log\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.061932 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061762    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkglq\" (UniqueName: \"kubernetes.io/projected/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-kube-api-access-vkglq\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.061932 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.061932 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061851    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-kubelet-config\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.061932 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-system-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.061932 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061905    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529ebf7f-7783-4575-a1fd-5c672e410f6a-host-slash\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.061932 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061931    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-kubelet\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061957    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-slash\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.061983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-script-lib\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmfr2\" (UniqueName: \"kubernetes.io/projected/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kube-api-access-qmfr2\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062091    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-k8s-cni-cncf-io\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-multus\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062148    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062195    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a046e81-b1ab-4da2-ac50-7d06f2c30668-tmp-dir\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-daemon-config\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-netns\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.062386 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062387    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-var-lib-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062425    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4gx\" (UniqueName: \"kubernetes.io/projected/83bc0294-07c9-4087-88cf-1cf891d84118-kube-api-access-wc4gx\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062479    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c365930-eba1-4476-8a6c-7efe851313a8-konnectivity-ca\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062515    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57050003-649e-4fad-ba51-9f1a534a859c-host\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062579    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-run\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062642    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062676    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062713    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvbj\" (UniqueName: \"kubernetes.io/projected/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-kube-api-access-jdvbj\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062740    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-etc-kubernetes\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppj6l\" (UniqueName: \"kubernetes.io/projected/b6d1b009-1219-4dba-97a5-32d9f88b1447-kube-api-access-ppj6l\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.062846 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062794    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/529ebf7f-7783-4575-a1fd-5c672e410f6a-iptables-alerter-script\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kctn\" (UniqueName: \"kubernetes.io/projected/529ebf7f-7783-4575-a1fd-5c672e410f6a-kube-api-access-9kctn\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062907    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57050003-649e-4fad-ba51-9f1a534a859c-serviceca\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062933    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-netns\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062958    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-systemd-units\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062981    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-etc-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.062998    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-ovn\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063023    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2d2w\" (UniqueName: \"kubernetes.io/projected/b88993de-a05b-40ff-a753-8b9949f5f6c4-kube-api-access-s2d2w\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063053    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p46b\" (UniqueName: \"kubernetes.io/projected/57050003-649e-4fad-ba51-9f1a534a859c-kube-api-access-9p46b\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063110    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-lib-modules\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063130    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-tmp\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063147    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-cnibin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063163    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-systemd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063188    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063212    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-config\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063236    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmb7\" (UniqueName: \"kubernetes.io/projected/3a046e81-b1ab-4da2-ac50-7d06f2c30668-kube-api-access-2xmb7\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063276    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.063355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063308    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-socket-dir-parent\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063333    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-log-socket\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-netd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063384    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-kubernetes\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063412    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-conf\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063475    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83bc0294-07c9-4087-88cf-1cf891d84118-ovn-node-metrics-cert\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063506    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063580    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c365930-eba1-4476-8a6c-7efe851313a8-agent-certs\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063626    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063655    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-device-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063686    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063718    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063754    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cnibin\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063778    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-host\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063795    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063812    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.064179 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063877    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063907    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-var-lib-kubelet\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-tuned\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-cni-binary-copy\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063969    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-bin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.063996    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-bin\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064011    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-systemd\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064043    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-sys\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064060    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064077    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064100    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-multus-certs\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-modprobe-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064138    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysconfig\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064153    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-kubelet\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064183    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-hostroot\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.064989 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.064255    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-conf-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.068819 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.068765    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal" event={"ID":"041057424ec306b79ef7363e62fcbac0","Type":"ContainerStarted","Data":"ec42e889039300d4ce8157ac0177cafa75e1397bb94c07394e66beaedbc228dd"}
Oct 30 16:22:12.070491 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.070429    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal" event={"ID":"a6861b6533e84603bf25f2e81fd4e489","Type":"ContainerStarted","Data":"7ab6d0bd18409d2c903badb6074cca41ba8bcf67965b2f5cdca4078b5ec9ad34"}
Oct 30 16:22:12.085635 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.085583    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:22:12.085635 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.085598    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 16:22:12.085867 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.085673    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 16:22:12.085867 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.085733    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 16:22:12.085867 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.085861    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 16:22:12.086023 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.085946    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 16:22:12.086023 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.085953    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 16:22:12.086023 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.085975    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 16:22:12.086139 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086060    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 16:22:12.086139 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086089    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 16:22:12.086139 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086092    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 16:22:12.086276 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086166    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 16:22:12.086276 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086243    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 16:22:12.086734 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086646    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 16:22:12.086734 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086718    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 16:22:12.086734 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086729    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:22:12.086954 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086751    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 16:22:12.086954 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.086865    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 16:22:12.087105 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087087    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 16:22:12.087172 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087147    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5n7vr\""
Oct 30 16:22:12.087172 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087165    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h4vt2\""
Oct 30 16:22:12.087307 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087175    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m9x9f\""
Oct 30 16:22:12.087307 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087200    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lf54z\""
Oct 30 16:22:12.087307 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087167    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 16:22:12.087307 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087234    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjhbz\""
Oct 30 16:22:12.087307 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087233    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tdd2t\""
Oct 30 16:22:12.087307 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087267    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kf5z\""
Oct 30 16:22:12.087527 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.087361    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.164939    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.164989    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-node-log\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165027    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkglq\" (UniqueName: \"kubernetes.io/projected/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-kube-api-access-vkglq\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165045    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165061    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-kubelet-config\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165075    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-system-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165090    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529ebf7f-7783-4575-a1fd-5c672e410f6a-host-slash\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165104    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-kubelet\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165124    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-slash\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-script-lib\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165161    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfr2\" (UniqueName: \"kubernetes.io/projected/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kube-api-access-qmfr2\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165184    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-k8s-cni-cncf-io\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165209    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-multus\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165245    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a046e81-b1ab-4da2-ac50-7d06f2c30668-tmp-dir\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-daemon-config\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.168255 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165299    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-netns\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-var-lib-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165335    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4gx\" (UniqueName: \"kubernetes.io/projected/83bc0294-07c9-4087-88cf-1cf891d84118-kube-api-access-wc4gx\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c365930-eba1-4476-8a6c-7efe851313a8-konnectivity-ca\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165373    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57050003-649e-4fad-ba51-9f1a534a859c-host\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165397    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-run\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165422    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165438    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvbj\" (UniqueName: \"kubernetes.io/projected/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-kube-api-access-jdvbj\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165479    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-etc-kubernetes\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165496    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppj6l\" (UniqueName: \"kubernetes.io/projected/b6d1b009-1219-4dba-97a5-32d9f88b1447-kube-api-access-ppj6l\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165512    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/529ebf7f-7783-4575-a1fd-5c672e410f6a-iptables-alerter-script\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165533    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kctn\" (UniqueName: \"kubernetes.io/projected/529ebf7f-7783-4575-a1fd-5c672e410f6a-kube-api-access-9kctn\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165552    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57050003-649e-4fad-ba51-9f1a534a859c-serviceca\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165571    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-netns\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165587    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-systemd-units\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-etc-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.169289 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165623    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-ovn\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165641    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2d2w\" (UniqueName: \"kubernetes.io/projected/b88993de-a05b-40ff-a753-8b9949f5f6c4-kube-api-access-s2d2w\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p46b\" (UniqueName: \"kubernetes.io/projected/57050003-649e-4fad-ba51-9f1a534a859c-kube-api-access-9p46b\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165704    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-lib-modules\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165724    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-tmp\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165743    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-cnibin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165773    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-systemd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165789    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165806    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-config\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmb7\" (UniqueName: \"kubernetes.io/projected/3a046e81-b1ab-4da2-ac50-7d06f2c30668-kube-api-access-2xmb7\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165845    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165867    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-socket-dir-parent\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165893    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-log-socket\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165920    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-netd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165945    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-kubernetes\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165971    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-conf\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.165992    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83bc0294-07c9-4087-88cf-1cf891d84118-ovn-node-metrics-cert\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.170113 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166008    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166024    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c365930-eba1-4476-8a6c-7efe851313a8-agent-certs\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166040    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166064    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-device-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166086    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166105    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166123    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cnibin\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-host\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166187    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166205    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166227    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166252    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-var-lib-kubelet\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-tuned\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-cni-binary-copy\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166322    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-bin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-bin\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166367    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-systemd\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.170974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166395    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-sys\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166422    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166448    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166479    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-multus-certs\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166505    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-modprobe-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166523    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysconfig\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166540    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-kubelet\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-hostroot\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-conf-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166614    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166640    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-dbus\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166661    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-env-overrides\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166679    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166698    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a046e81-b1ab-4da2-ac50-7d06f2c30668-hosts-file\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.166810    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a046e81-b1ab-4da2-ac50-7d06f2c30668-hosts-file\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167060    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167153    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-var-lib-kubelet\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.171930 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167350    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-etc-kubernetes\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167470    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57050003-649e-4fad-ba51-9f1a534a859c-host\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167163    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167585    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-node-log\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167604    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167653    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-netns\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167661    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-hostroot\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.167753    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-lib-modules\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.167834    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168052    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-netns\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168100    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168141    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-systemd-units\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168173    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-var-lib-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168175    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-cnibin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168263    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168214    2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168302    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168314    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-cni-binary-copy\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168340    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.172696 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168380    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-socket-dir-parent\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168416    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-log-socket\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168451    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-netd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.168470    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret podName:536629a7-e02f-4d49-a9de-cccd0e39132e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:12.667885009 +0000 UTC m=+3.054763920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret") pod "global-pull-secret-syncer-x4pp5" (UID: "536629a7-e02f-4d49-a9de-cccd0e39132e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168516    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168535    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-kubernetes\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168566    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-run\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168605    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-etc-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168645    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-ovn\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168660    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cnibin\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168704    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168928    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-daemon-config\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.168984    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a046e81-b1ab-4da2-ac50-7d06f2c30668-tmp-dir\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.169183    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-systemd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.169228    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-conf-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.169289    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-slash\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.169762    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/529ebf7f-7783-4575-a1fd-5c672e410f6a-iptables-alerter-script\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.173430 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.169895    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-config\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.169931    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-multus\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.169951    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-k8s-cni-cncf-io\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170020    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-host\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170065    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-kubelet-config\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.170094    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.170145    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:12.67012858 +0000 UTC m=+3.057007480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170149    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170182    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-multus-certs\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170223    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-system-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170300    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-modprobe-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170337    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-bin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170023    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-dbus\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170422    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysconfig\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170437    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-conf\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170463    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529ebf7f-7783-4575-a1fd-5c672e410f6a-host-slash\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170509    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-device-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170533    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57050003-649e-4fad-ba51-9f1a534a859c-serviceca\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.174036 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170735    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-systemd\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170780    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-bin\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170819    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170827    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-kubelet\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170850    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-kubelet\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170866    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-sys\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.170920    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.171403    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c365930-eba1-4476-8a6c-7efe851313a8-konnectivity-ca\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.171781    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.171929    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-script-lib\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.172431    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.172897    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-env-overrides\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.173177    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83bc0294-07c9-4087-88cf-1cf891d84118-ovn-node-metrics-cert\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.173369    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-tmp\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.174816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.174226    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c365930-eba1-4476-8a6c-7efe851313a8-agent-certs\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:12.184017 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.183920    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-tuned\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.184218 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.184190    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.191840 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.191812    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppj6l\" (UniqueName: \"kubernetes.io/projected/b6d1b009-1219-4dba-97a5-32d9f88b1447-kube-api-access-ppj6l\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.192915 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.192862    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkglq\" (UniqueName: \"kubernetes.io/projected/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-kube-api-access-vkglq\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.194599 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.194575    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:22:12.194703 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.194606    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:22:12.194703 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.194621    2570 projected.go:196] Error preparing data for projected volume kube-api-access-4qgv9 for pod openshift-network-diagnostics/network-check-target-htgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:12.194703 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.194685    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9 podName:bea838c0-50c6-4c52-a0ff-e1e019022677 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:12.694664723 +0000 UTC m=+3.081543618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4qgv9" (UniqueName: "kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9") pod "network-check-target-htgvf" (UID: "bea838c0-50c6-4c52-a0ff-e1e019022677") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:12.196853 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.196795    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvbj\" (UniqueName: \"kubernetes.io/projected/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-kube-api-access-jdvbj\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.196853 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.196794    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmb7\" (UniqueName: \"kubernetes.io/projected/3a046e81-b1ab-4da2-ac50-7d06f2c30668-kube-api-access-2xmb7\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.196977 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.196907    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p46b\" (UniqueName: \"kubernetes.io/projected/57050003-649e-4fad-ba51-9f1a534a859c-kube-api-access-9p46b\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.198888 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.198863    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2d2w\" (UniqueName: \"kubernetes.io/projected/b88993de-a05b-40ff-a753-8b9949f5f6c4-kube-api-access-s2d2w\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:12.203968 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.203807    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfr2\" (UniqueName: \"kubernetes.io/projected/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kube-api-access-qmfr2\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.204272 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.204216    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kctn\" (UniqueName: \"kubernetes.io/projected/529ebf7f-7783-4575-a1fd-5c672e410f6a-kube-api-access-9kctn\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.209831 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.209741    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4gx\" (UniqueName: \"kubernetes.io/projected/83bc0294-07c9-4087-88cf-1cf891d84118-kube-api-access-wc4gx\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.293097 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.293060    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k545t"
Oct 30 16:22:12.295277 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.295243    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:12.299982 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.299829    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:12.306015 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:12.305984    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d1b009_1219_4dba_97a5_32d9f88b1447.slice/crio-85571300fa7ff721d661ab49ad448ae4f261d35b1cb114eaa19e194e6ad60388 WatchSource:0}: Error finding container 85571300fa7ff721d661ab49ad448ae4f261d35b1cb114eaa19e194e6ad60388: Status 404 returned error can't find the container with id 85571300fa7ff721d661ab49ad448ae4f261d35b1cb114eaa19e194e6ad60388
Oct 30 16:22:12.311629 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:12.311593    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b89a9fa_bf4d_44f6_9331_22a216a06f6d.slice/crio-b13f2f43a31ccadae64086dd32f85505f60d7a370b57b1f32042ca27f351758b WatchSource:0}: Error finding container b13f2f43a31ccadae64086dd32f85505f60d7a370b57b1f32042ca27f351758b: Status 404 returned error can't find the container with id b13f2f43a31ccadae64086dd32f85505f60d7a370b57b1f32042ca27f351758b
Oct 30 16:22:12.312475 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:12.312449    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529ebf7f_7783_4575_a1fd_5c672e410f6a.slice/crio-0b0dc445a7833cd606d2b04a1f8258674bb82b15dc1fabf3bf6e65de59621145 WatchSource:0}: Error finding container 0b0dc445a7833cd606d2b04a1f8258674bb82b15dc1fabf3bf6e65de59621145: Status 404 returned error can't find the container with id 0b0dc445a7833cd606d2b04a1f8258674bb82b15dc1fabf3bf6e65de59621145
Oct 30 16:22:12.319535 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.319498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:12.326879 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.326694    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:12.328557 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.328431    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:12.331644 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:12.331617    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57050003_649e_4fad_ba51_9f1a534a859c.slice/crio-4b94dc792042463d6e9ee676b30c0d573c34fa773a130179072cfaa6230e158f WatchSource:0}: Error finding container 4b94dc792042463d6e9ee676b30c0d573c34fa773a130179072cfaa6230e158f: Status 404 returned error can't find the container with id 4b94dc792042463d6e9ee676b30c0d573c34fa773a130179072cfaa6230e158f
Oct 30 16:22:12.331742 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.331643    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:12.338574 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:12.338535    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bc0294_07c9_4087_88cf_1cf891d84118.slice/crio-1d10232299381461a1bf6b7e96c8cdce29e34cd1a0cc1d1638e92f602acea66a WatchSource:0}: Error finding container 1d10232299381461a1bf6b7e96c8cdce29e34cd1a0cc1d1638e92f602acea66a: Status 404 returned error can't find the container with id 1d10232299381461a1bf6b7e96c8cdce29e34cd1a0cc1d1638e92f602acea66a
Oct 30 16:22:12.341257 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:12.341207    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bd08bb_58ea_4508_8e99_f4a1d9b552ad.slice/crio-739e7b672f8f7b7f9f3892df773873631e46ca8c977145c1750159e3feb12243 WatchSource:0}: Error finding container 739e7b672f8f7b7f9f3892df773873631e46ca8c977145c1750159e3feb12243: Status 404 returned error can't find the container with id 739e7b672f8f7b7f9f3892df773873631e46ca8c977145c1750159e3feb12243
Oct 30 16:22:12.345929 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.345878    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:12.348416 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:12.347605    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a046e81_b1ab_4da2_ac50_7d06f2c30668.slice/crio-6fa79b6ad1e3fd8ceac59474a0e16df925ed4b3beb27b4ef5f7b192bb793cfbc WatchSource:0}: Error finding container 6fa79b6ad1e3fd8ceac59474a0e16df925ed4b3beb27b4ef5f7b192bb793cfbc: Status 404 returned error can't find the container with id 6fa79b6ad1e3fd8ceac59474a0e16df925ed4b3beb27b4ef5f7b192bb793cfbc
Oct 30 16:22:12.355169 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.355147    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:12.357958 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:12.357928    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c365930_eba1_4476_8a6c_7efe851313a8.slice/crio-11df67c1bef2bea44c38376b183f4ae806638d276b0cb84089db06854c152b23 WatchSource:0}: Error finding container 11df67c1bef2bea44c38376b183f4ae806638d276b0cb84089db06854c152b23: Status 404 returned error can't find the container with id 11df67c1bef2bea44c38376b183f4ae806638d276b0cb84089db06854c152b23
Oct 30 16:22:12.669929 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.669888    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:12.670116 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.670042    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:12.670188 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.670123    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret podName:536629a7-e02f-4d49-a9de-cccd0e39132e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:13.670101066 +0000 UTC m=+4.056979974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret") pod "global-pull-secret-syncer-x4pp5" (UID: "536629a7-e02f-4d49-a9de-cccd0e39132e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:12.770660 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.770603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:12.770804 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:12.770695    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:12.770876 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.770844    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:12.770933 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.770909    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:13.770889459 +0000 UTC m=+4.157768369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:12.771379 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.771357    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:22:12.771479 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.771390    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:22:12.771479 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.771401    2570 projected.go:196] Error preparing data for projected volume kube-api-access-4qgv9 for pod openshift-network-diagnostics/network-check-target-htgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:12.771479 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:12.771453    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9 podName:bea838c0-50c6-4c52-a0ff-e1e019022677 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:13.771436196 +0000 UTC m=+4.158315096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qgv9" (UniqueName: "kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9") pod "network-check-target-htgvf" (UID: "bea838c0-50c6-4c52-a0ff-e1e019022677") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:13.030593 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.030476    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:12 +0000 UTC" deadline="2028-05-19 06:36:05.928172872 +0000 UTC"
Oct 30 16:22:13.030593 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.030544    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13598h13m52.897633419s"
Oct 30 16:22:13.086352 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.086312    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"1d10232299381461a1bf6b7e96c8cdce29e34cd1a0cc1d1638e92f602acea66a"}
Oct 30 16:22:13.098095 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.098053    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rc86k" event={"ID":"57050003-649e-4fad-ba51-9f1a534a859c","Type":"ContainerStarted","Data":"4b94dc792042463d6e9ee676b30c0d573c34fa773a130179072cfaa6230e158f"}
Oct 30 16:22:13.112446 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.100734    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jwpjd" event={"ID":"529ebf7f-7783-4575-a1fd-5c672e410f6a","Type":"ContainerStarted","Data":"0b0dc445a7833cd606d2b04a1f8258674bb82b15dc1fabf3bf6e65de59621145"}
Oct 30 16:22:13.112446 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.104931    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jv959" event={"ID":"8b89a9fa-bf4d-44f6-9331-22a216a06f6d","Type":"ContainerStarted","Data":"b13f2f43a31ccadae64086dd32f85505f60d7a370b57b1f32042ca27f351758b"}
Oct 30 16:22:13.112446 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.108233    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal" event={"ID":"041057424ec306b79ef7363e62fcbac0","Type":"ContainerStarted","Data":"8795e8d324639119a918af97447a511832d64bde9c9de9380e0eb5036f22073a"}
Oct 30 16:22:13.113442 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.113334    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg" event={"ID":"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8","Type":"ContainerStarted","Data":"694805d01bf18c4c078132aea8a4e991a4f2e6edede8e355c9adb01cf2b7b983"}
Oct 30 16:22:13.116267 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.116168    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k545t" event={"ID":"b6d1b009-1219-4dba-97a5-32d9f88b1447","Type":"ContainerStarted","Data":"85571300fa7ff721d661ab49ad448ae4f261d35b1cb114eaa19e194e6ad60388"}
Oct 30 16:22:13.122033 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.121410    2570 generic.go:356] "Generic (PLEG): container finished" podID="a6861b6533e84603bf25f2e81fd4e489" containerID="d9057d79573e6d02c4e65dae9a6ef161f4c9c6affe419c7d837f7d028f62bb4b" exitCode=0
Oct 30 16:22:13.122033 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.121473    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal" event={"ID":"a6861b6533e84603bf25f2e81fd4e489","Type":"ContainerDied","Data":"d9057d79573e6d02c4e65dae9a6ef161f4c9c6affe419c7d837f7d028f62bb4b"}
Oct 30 16:22:13.127818 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.126436    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6fh2" event={"ID":"3c365930-eba1-4476-8a6c-7efe851313a8","Type":"ContainerStarted","Data":"11df67c1bef2bea44c38376b183f4ae806638d276b0cb84089db06854c152b23"}
Oct 30 16:22:13.132586 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.132491    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2lxwn" event={"ID":"3a046e81-b1ab-4da2-ac50-7d06f2c30668","Type":"ContainerStarted","Data":"6fa79b6ad1e3fd8ceac59474a0e16df925ed4b3beb27b4ef5f7b192bb793cfbc"}
Oct 30 16:22:13.159118 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.159007    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pz9z8" event={"ID":"31bd08bb-58ea-4508-8e99-f4a1d9b552ad","Type":"ContainerStarted","Data":"739e7b672f8f7b7f9f3892df773873631e46ca8c977145c1750159e3feb12243"}
Oct 30 16:22:13.195030 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.194752    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal" podStartSLOduration=2.194728724 podStartE2EDuration="2.194728724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:13.153400427 +0000 UTC m=+3.540279335" watchObservedRunningTime="2026-10-30 16:22:13.194728724 +0000 UTC m=+3.581607633"
Oct 30 16:22:13.678247 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.678154    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:13.678473 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:13.678379    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:13.678473 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:13.678456    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret podName:536629a7-e02f-4d49-a9de-cccd0e39132e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:15.678436794 +0000 UTC m=+6.065315685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret") pod "global-pull-secret-syncer-x4pp5" (UID: "536629a7-e02f-4d49-a9de-cccd0e39132e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:13.778752 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.778712    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:13.778929 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:13.778796    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:13.778973 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:13.778957    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:13.779038 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:13.779025    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:15.779005946 +0000 UTC m=+6.165884835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:13.779506 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:13.779478    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:22:13.779506 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:13.779507    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:22:13.779611 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:13.779521    2570 projected.go:196] Error preparing data for projected volume kube-api-access-4qgv9 for pod openshift-network-diagnostics/network-check-target-htgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:13.779611 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:13.779570    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9 podName:bea838c0-50c6-4c52-a0ff-e1e019022677 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:15.77955325 +0000 UTC m=+6.166432135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qgv9" (UniqueName: "kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9") pod "network-check-target-htgvf" (UID: "bea838c0-50c6-4c52-a0ff-e1e019022677") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:14.030896 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:14.030784    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:12 +0000 UTC" deadline="2028-05-19 13:22:29.509399506 +0000 UTC"
Oct 30 16:22:14.030896 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:14.030853    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13605h0m15.478551637s"
Oct 30 16:22:14.067838 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:14.067334    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:14.067838 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:14.067475    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:14.068074 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:14.067894    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:14.068074 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:14.067989    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:14.068074 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:14.068054    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:14.068229 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:14.068125    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:14.184412 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:14.184380    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal" event={"ID":"a6861b6533e84603bf25f2e81fd4e489","Type":"ContainerStarted","Data":"cc595e8d0c269d2f1a536e41dc3ae8797dfc1d5c3d9103f4755f75abd15927c9"}
Oct 30 16:22:14.228599 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:14.228534    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal" podStartSLOduration=3.228510814 podStartE2EDuration="3.228510814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:14.22775903 +0000 UTC m=+4.614637940" watchObservedRunningTime="2026-10-30 16:22:14.228510814 +0000 UTC m=+4.615389724"
Oct 30 16:22:15.697707 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:15.697126    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:15.697707 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:15.697304    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:15.697707 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:15.697369    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret podName:536629a7-e02f-4d49-a9de-cccd0e39132e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:19.697351152 +0000 UTC m=+10.084230032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret") pod "global-pull-secret-syncer-x4pp5" (UID: "536629a7-e02f-4d49-a9de-cccd0e39132e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:15.799265 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:15.798309    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:15.799265 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:15.798444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:15.799265 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:15.798607    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:22:15.799265 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:15.798632    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:22:15.799265 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:15.798645    2570 projected.go:196] Error preparing data for projected volume kube-api-access-4qgv9 for pod openshift-network-diagnostics/network-check-target-htgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:15.799265 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:15.798742    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9 podName:bea838c0-50c6-4c52-a0ff-e1e019022677 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:19.798722921 +0000 UTC m=+10.185601820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qgv9" (UniqueName: "kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9") pod "network-check-target-htgvf" (UID: "bea838c0-50c6-4c52-a0ff-e1e019022677") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:15.799265 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:15.799181    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:15.799265 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:15.799239    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:19.799227384 +0000 UTC m=+10.186106264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:16.066476 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:16.066297    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:16.066476 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:16.066432    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:16.066476 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:16.066469    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:16.066683 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:16.066549    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:16.066683 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:16.066583    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:16.066683 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:16.066643    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:16.221493 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:16.221460    2570 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="e4daa5fda1466410e1ed370d6888de951bd2f8fe3cfd932afd0995b966dfa412" exitCode=0
Oct 30 16:22:16.221675 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:16.221510    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jv959" event={"ID":"8b89a9fa-bf4d-44f6-9331-22a216a06f6d","Type":"ContainerDied","Data":"e4daa5fda1466410e1ed370d6888de951bd2f8fe3cfd932afd0995b966dfa412"}
Oct 30 16:22:17.231277 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:17.230550    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6fh2" event={"ID":"3c365930-eba1-4476-8a6c-7efe851313a8","Type":"ContainerStarted","Data":"4d3b1ca6d4906d5a948efcc6cef686f83fe84f974250acde3c9023b0702a3a48"}
Oct 30 16:22:18.066202 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:18.066162    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:18.066365 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:18.066303    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:18.066365 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:18.066311    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:18.066365 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:18.066344    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:18.066532 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:18.066401    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:18.066532 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:18.066484    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:18.235851 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:18.235766    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rc86k" event={"ID":"57050003-649e-4fad-ba51-9f1a534a859c","Type":"ContainerStarted","Data":"2faf5ae470d640c7ef17e6cc1ee682e5ff03b91189a6bce10c0be970b4954cef"}
Oct 30 16:22:18.240522 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:18.240431    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg" event={"ID":"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8","Type":"ContainerStarted","Data":"a54d712de9e7c1148441cdfdccc6e4e4dd28fdda68cc9d2667066d81c14d4a31"}
Oct 30 16:22:18.272180 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:18.271777    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j6fh2" podStartSLOduration=4.097312127 podStartE2EDuration="8.271757772s" totalImagesPullingTime="4.174445645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:17.268558817 +0000 UTC m=+7.655437728" watchObservedRunningTime="2026-10-30 16:22:18.271757772 +0000 UTC m=+8.658636674"
Oct 30 16:22:19.592402 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:19.592091    2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 16:22:19.731109 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:19.731019    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:19.731273 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:19.731170    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:19.731273 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:19.731254    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret podName:536629a7-e02f-4d49-a9de-cccd0e39132e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:27.731235674 +0000 UTC m=+18.118114569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret") pod "global-pull-secret-syncer-x4pp5" (UID: "536629a7-e02f-4d49-a9de-cccd0e39132e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:19.831358 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:19.831318    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:19.831532 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:19.831409    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:19.831594 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:19.831557    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:19.831649 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:19.831621    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:27.831601887 +0000 UTC m=+18.218480772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:19.832058 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:19.832039    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:22:19.832117 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:19.832064    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:22:19.832117 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:19.832078    2570 projected.go:196] Error preparing data for projected volume kube-api-access-4qgv9 for pod openshift-network-diagnostics/network-check-target-htgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:19.832228 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:19.832126    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9 podName:bea838c0-50c6-4c52-a0ff-e1e019022677 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:27.832112735 +0000 UTC m=+18.218991620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qgv9" (UniqueName: "kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9") pod "network-check-target-htgvf" (UID: "bea838c0-50c6-4c52-a0ff-e1e019022677") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:20.008472 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.007498    2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:22:19.592109206Z","UUID":"b03d1fd8-453c-40b6-9de5-a715f1b4e665","Handler":null,"Name":"","Endpoint":""}
Oct 30 16:22:20.016581 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.016546    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 16:22:20.016581 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.016582    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 16:22:20.067093 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.065420    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:20.067093 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.065464    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:20.067093 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.065420    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:20.067093 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:20.065550    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:20.067093 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:20.065637    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:20.067093 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:20.065711    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:20.251727 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.251624    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg" event={"ID":"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8","Type":"ContainerStarted","Data":"11f686e6cdcf75a5ba5fb2f1e1895d06696ff3a5f5efbcc64b8872dce59f882d"}
Oct 30 16:22:20.254328 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.253584    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k545t" event={"ID":"b6d1b009-1219-4dba-97a5-32d9f88b1447","Type":"ContainerStarted","Data":"381c7987029fc4512544999c7297d903c59df19065c1016aec6cb9052f2aa77c"}
Oct 30 16:22:20.294331 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.294164    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rc86k" podStartSLOduration=5.484898031 podStartE2EDuration="10.294144891s" totalImagesPullingTime="4.80924686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:18.272586222 +0000 UTC m=+8.659465124" watchObservedRunningTime="2026-10-30 16:22:20.294144891 +0000 UTC m=+10.681023794"
Oct 30 16:22:20.294496 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.294442    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-k545t" podStartSLOduration=2.5127790709999998 podStartE2EDuration="10.294432067s" totalImagesPullingTime="7.781652996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:20.293519031 +0000 UTC m=+10.680397935" watchObservedRunningTime="2026-10-30 16:22:20.294432067 +0000 UTC m=+10.681310970"
Oct 30 16:22:20.954137 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.954063    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:20.954533 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:20.954517    2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:21.257771 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:21.257733    2570 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="4d08ef3e4528d6fa9964545a1656bb0749d4d1cd7fe8ed14d19a73dd76f5a88c" exitCode=0
Oct 30 16:22:21.257942 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:21.257818    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jv959" event={"ID":"8b89a9fa-bf4d-44f6-9331-22a216a06f6d","Type":"ContainerDied","Data":"4d08ef3e4528d6fa9964545a1656bb0749d4d1cd7fe8ed14d19a73dd76f5a88c"}
Oct 30 16:22:21.260513 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:21.260486    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg" event={"ID":"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8","Type":"ContainerStarted","Data":"16ace576a4d873a846bc62329e861158443536e82278089cc128a31d1a5801b2"}
Oct 30 16:22:21.263316 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:21.263287    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2lxwn" event={"ID":"3a046e81-b1ab-4da2-ac50-7d06f2c30668","Type":"ContainerStarted","Data":"7fa8177f148bb303fc6c7a696dba881db6988872eca793f6104dfb12393cf10b"}
Oct 30 16:22:21.263638 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:21.263615    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:21.263760 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:21.263677    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:21.335564 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:21.335512    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2lxwn" podStartSLOduration=3.211888449 podStartE2EDuration="11.335492349s" totalImagesPullingTime="8.1236039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:21.335491352 +0000 UTC m=+11.722370254" watchObservedRunningTime="2026-10-30 16:22:21.335492349 +0000 UTC m=+11.722371252"
Oct 30 16:22:21.373723 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:21.373492    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg" podStartSLOduration=3.39084597 podStartE2EDuration="11.373474478s" totalImagesPullingTime="7.982628508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:21.37303548 +0000 UTC m=+11.759914380" watchObservedRunningTime="2026-10-30 16:22:21.373474478 +0000 UTC m=+11.760353383"
Oct 30 16:22:22.064850 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:22.064813    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:22.064850 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:22.064845    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:22.065440 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:22.064818    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:22.065440 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:22.064951    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:22.065440 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:22.065040    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:22.065440 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:22.065126    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:22.265622 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:22.265588    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jwpjd" event={"ID":"529ebf7f-7783-4575-a1fd-5c672e410f6a","Type":"ContainerStarted","Data":"8fb1260254c571e25a6820f3724b890a60b8c7f00c8640a333abae561c12d3d3"}
Oct 30 16:22:22.267974 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:22.267948    2570 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="a5a4001ea523983a4277f5b5d2dc1e5f5fc12fa61e15ac834de1da810594e72d" exitCode=0
Oct 30 16:22:22.268103 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:22.268050    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jv959" event={"ID":"8b89a9fa-bf4d-44f6-9331-22a216a06f6d","Type":"ContainerDied","Data":"a5a4001ea523983a4277f5b5d2dc1e5f5fc12fa61e15ac834de1da810594e72d"}
Oct 30 16:22:22.341145 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:22.341046    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jwpjd" podStartSLOduration=4.309929336 podStartE2EDuration="12.341029512s" totalImagesPullingTime="8.031100176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:22.301482 +0000 UTC m=+12.688360906" watchObservedRunningTime="2026-10-30 16:22:22.341029512 +0000 UTC m=+12.727908415"
Oct 30 16:22:23.271900 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:23.271861    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pz9z8" event={"ID":"31bd08bb-58ea-4508-8e99-f4a1d9b552ad","Type":"ContainerStarted","Data":"ee7d8f1f96789c326675e04187cb02b8abf216634249b783b107b70f33200704"}
Oct 30 16:22:23.274596 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:23.274552    2570 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="5f8c7f6d053ed4907559fc1422ec4ca11d334a669f9ae88899776588a1b371b3" exitCode=0
Oct 30 16:22:23.274717 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:23.274660    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jv959" event={"ID":"8b89a9fa-bf4d-44f6-9331-22a216a06f6d","Type":"ContainerDied","Data":"5f8c7f6d053ed4907559fc1422ec4ca11d334a669f9ae88899776588a1b371b3"}
Oct 30 16:22:23.311158 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:23.311106    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pz9z8" podStartSLOduration=3.106339662 podStartE2EDuration="13.311089504s" totalImagesPullingTime="10.204749842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:23.31092934 +0000 UTC m=+13.697808243" watchObservedRunningTime="2026-10-30 16:22:23.311089504 +0000 UTC m=+13.697968408"
Oct 30 16:22:24.064072 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:24.064031    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:24.064264 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:24.064041    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:24.064264 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:24.064206    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:24.064264 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:24.064203    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:24.064424 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:24.064351    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:24.064469 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:24.064423    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:26.064609 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:26.064400    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:26.065166 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:26.064400    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:26.065166 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:26.064713    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:26.065166 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:26.064808    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:26.065166 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:26.064444    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:26.065166 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:26.064909    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:26.283414 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:26.283380    2570 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="8b8d7de3cbf5daab09f07d87d29fc078031ca9c1b737e058b7f96d3a03137a13" exitCode=0
Oct 30 16:22:26.283558 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:26.283428    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jv959" event={"ID":"8b89a9fa-bf4d-44f6-9331-22a216a06f6d","Type":"ContainerDied","Data":"8b8d7de3cbf5daab09f07d87d29fc078031ca9c1b737e058b7f96d3a03137a13"}
Oct 30 16:22:27.289084 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.288855    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"6e0589ba1b3cb0e084a2ea2fe649ac466d9e02f7f48a1ce6b00ef316b9b48109"}
Oct 30 16:22:27.289557 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.289134    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"babfea3c55f4886856716ad0787863a715c331336fd4952477fa2b11e0490efe"}
Oct 30 16:22:27.289557 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.289206    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"3a536d54505cc55f4c0022096d6e25612d7398a25dd1d9969e2b2a9c3c22d84b"}
Oct 30 16:22:27.289557 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.289254    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"14fa8a9a3b79684dcbea46e0638afc03f3cae743916172139a8e9eb75d237b39"}
Oct 30 16:22:27.289557 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.289299    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"e5dc9d5d15f96b0b15eaccf407f804fbfdb784f82245dff2c679f33a93e3dd32"}
Oct 30 16:22:27.289557 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.289347    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"9231e42b357e7735ef7b457f898fe95c0d60edb1b3bedee7cc616ac4c681eeee"}
Oct 30 16:22:27.295074 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.295046    2570 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="7cbb50eaac6f92c79b2854d2ef9e079b960027b902def012926da6f410b3b279" exitCode=0
Oct 30 16:22:27.295400 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.295092    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jv959" event={"ID":"8b89a9fa-bf4d-44f6-9331-22a216a06f6d","Type":"ContainerDied","Data":"7cbb50eaac6f92c79b2854d2ef9e079b960027b902def012926da6f410b3b279"}
Oct 30 16:22:27.796175 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.796130    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:27.796373 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:27.796262    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:27.796373 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:27.796314    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret podName:536629a7-e02f-4d49-a9de-cccd0e39132e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:43.796302226 +0000 UTC m=+34.183181106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret") pod "global-pull-secret-syncer-x4pp5" (UID: "536629a7-e02f-4d49-a9de-cccd0e39132e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 16:22:27.896578 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.896542    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:27.896730 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:27.896593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:27.896730 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:27.896681    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:27.896730 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:27.896691    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 16:22:27.896730 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:27.896705    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 16:22:27.896730 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:27.896716    2570 projected.go:196] Error preparing data for projected volume kube-api-access-4qgv9 for pod openshift-network-diagnostics/network-check-target-htgvf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:27.896917 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:27.896741    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:43.896725008 +0000 UTC m=+34.283603889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 16:22:27.896917 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:27.896756    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9 podName:bea838c0-50c6-4c52-a0ff-e1e019022677 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:43.896749592 +0000 UTC m=+34.283628472 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4qgv9" (UniqueName: "kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9") pod "network-check-target-htgvf" (UID: "bea838c0-50c6-4c52-a0ff-e1e019022677") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 16:22:28.064487 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:28.064399    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:28.064611 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:28.064399    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:28.064611 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:28.064534    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:28.064611 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:28.064531    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:28.064709 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:28.064608    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:28.064709 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:28.064689    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:28.299926 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:28.299896    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jv959" event={"ID":"8b89a9fa-bf4d-44f6-9331-22a216a06f6d","Type":"ContainerStarted","Data":"a38459a8dea107e36426bcf0c953ee64f71f91d99fa37f2ef341c18cddbc50ee"}
Oct 30 16:22:28.341461 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:28.341370    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jv959" podStartSLOduration=7.646569533 podStartE2EDuration="18.341356842s" totalImagesPullingTime="10.694787309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:28.340724437 +0000 UTC m=+18.727603323" watchObservedRunningTime="2026-10-30 16:22:28.341356842 +0000 UTC m=+18.728235744"
Oct 30 16:22:29.306199 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:29.306152    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"c27c0b1a230b728a41001f22ae84cd403f752fc99c5944e192b5509599bec7b2"}
Oct 30 16:22:30.065118 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:30.065080    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:30.065298 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:30.065206    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:30.065298 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:30.065233    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:30.065436 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:30.065211    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:30.065436 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:30.065332    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:30.065436 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:30.065411    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:32.064314 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:32.064277    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:32.064945 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:32.064300    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:32.064945 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:32.064394    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:32.064945 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:32.064487    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:32.064945 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:32.064532    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:32.064945 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:32.064614    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:32.315812 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:32.315743    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66645" event={"ID":"83bc0294-07c9-4087-88cf-1cf891d84118","Type":"ContainerStarted","Data":"75b27bc3ea5eb96ed350ff37f5fc86b38568384daae97236f4632a94f1c2bc81"}
Oct 30 16:22:32.316067 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:32.316050    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:32.331758 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:32.331605    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:32.395569 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:32.395519    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-66645" podStartSLOduration=7.833695202 podStartE2EDuration="22.395506274s" totalImagesPullingTime="14.561811072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:32.394998104 +0000 UTC m=+22.781877007" watchObservedRunningTime="2026-10-30 16:22:32.395506274 +0000 UTC m=+22.782385209"
Oct 30 16:22:33.318390 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.318364    2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 16:22:33.318812 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.318788    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:33.337010 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.336970    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:33.450623 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.450534    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4pp5"]
Oct 30 16:22:33.450855 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.450673    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:33.450855 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:33.450758    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:33.454306 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.454273    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pv5p7"]
Oct 30 16:22:33.454437 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.454401    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:33.454539 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:33.454516    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:33.454878 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.454858    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-htgvf"]
Oct 30 16:22:33.454956 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:33.454931    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:33.455018 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:33.454998    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:34.320886 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:34.320858    2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 16:22:35.064199 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:35.064144    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:35.064199 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:35.064164    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:35.064412 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:35.064227    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:35.064412 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:35.064301    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:35.064412 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:35.064356    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:35.064510 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:35.064428    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:35.323314 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:35.323246    2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 16:22:37.064381 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:37.064350    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:37.064933 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:37.064387    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:37.064933 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:37.064468    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:37.064933 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:37.064484    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-htgvf" podUID="bea838c0-50c6-4c52-a0ff-e1e019022677"
Oct 30 16:22:37.064933 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:37.064532    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4pp5" podUID="536629a7-e02f-4d49-a9de-cccd0e39132e"
Oct 30 16:22:37.064933 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:37.064587    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pv5p7" podUID="b88993de-a05b-40ff-a753-8b9949f5f6c4"
Oct 30 16:22:37.692457 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:37.692372    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:37.692620 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:37.692556    2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 16:22:37.702889 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:37.702863    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:38.844912 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.844715    2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeReady"
Oct 30 16:22:38.845281 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.844964    2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 16:22:38.909518 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.909484    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ts8b4"]
Oct 30 16:22:38.911848 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.911829    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:38.912355 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.912233    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fszqg"]
Oct 30 16:22:38.914786 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.914766    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:38.932466 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.932446    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 16:22:38.932600 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.932582    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 16:22:38.932600 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.932585    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-65hn8\""
Oct 30 16:22:38.934276 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.934263    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 16:22:38.939969 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.939946    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ts8b4"]
Oct 30 16:22:38.966150 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.966116    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tjs\" (UniqueName: \"kubernetes.io/projected/b8853655-ace7-46f7-89f4-8be79930cffa-kube-api-access-n2tjs\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:38.966150 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.966155    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8853655-ace7-46f7-89f4-8be79930cffa-cert\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:38.966392 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.966179    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe1d12e-6cb7-487f-a352-590c8fcf7775-config-volume\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:38.966392 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.966215    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe1d12e-6cb7-487f-a352-590c8fcf7775-tmp-dir\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:38.966392 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.966272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46dw\" (UniqueName: \"kubernetes.io/projected/cfe1d12e-6cb7-487f-a352-590c8fcf7775-kube-api-access-m46dw\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:38.966546 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.966509    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:38.978552 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:38.978522    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fszqg"]
Oct 30 16:22:39.026960 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.026925    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 16:22:39.027222 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.027207    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z9zpr\""
Oct 30 16:22:39.027733 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.027721    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 16:22:39.064343 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.064308    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:39.064343 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.064335    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:39.064529 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.064357    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:39.067302 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.067278    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:39.067414 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.067326    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tjs\" (UniqueName: \"kubernetes.io/projected/b8853655-ace7-46f7-89f4-8be79930cffa-kube-api-access-n2tjs\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:39.067414 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.067346    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8853655-ace7-46f7-89f4-8be79930cffa-cert\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:39.067534 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:39.067517    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:22:39.067592 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:39.067582    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls podName:cfe1d12e-6cb7-487f-a352-590c8fcf7775 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.567562629 +0000 UTC m=+29.954441513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls") pod "dns-default-ts8b4" (UID: "cfe1d12e-6cb7-487f-a352-590c8fcf7775") : secret "dns-default-metrics-tls" not found
Oct 30 16:22:39.067592 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.067521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe1d12e-6cb7-487f-a352-590c8fcf7775-config-volume\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:39.067698 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.067621    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe1d12e-6cb7-487f-a352-590c8fcf7775-tmp-dir\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:39.067698 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.067650    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46dw\" (UniqueName: \"kubernetes.io/projected/cfe1d12e-6cb7-487f-a352-590c8fcf7775-kube-api-access-m46dw\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:39.067956 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.067936    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe1d12e-6cb7-487f-a352-590c8fcf7775-tmp-dir\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:39.068082 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.068064    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe1d12e-6cb7-487f-a352-590c8fcf7775-config-volume\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:39.071514 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.071497    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8853655-ace7-46f7-89f4-8be79930cffa-cert\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:39.092531 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.092511    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tjs\" (UniqueName: \"kubernetes.io/projected/b8853655-ace7-46f7-89f4-8be79930cffa-kube-api-access-n2tjs\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:39.092617 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.092591    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46dw\" (UniqueName: \"kubernetes.io/projected/cfe1d12e-6cb7-487f-a352-590c8fcf7775-kube-api-access-m46dw\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:39.127316 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.127238    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 16:22:39.127528 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.127509    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hpqxb\""
Oct 30 16:22:39.127586 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.127511    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vhqb4\""
Oct 30 16:22:39.127586 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.127562    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 16:22:39.127586 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.127565    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9txpl\""
Oct 30 16:22:39.127816 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.127806    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 16:22:39.127986 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.127974    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 16:22:39.229283 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.229247    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:39.537857 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.537826    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fszqg"]
Oct 30 16:22:39.541645 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:39.541617    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8853655_ace7_46f7_89f4_8be79930cffa.slice/crio-f23c0dd1b15c4d196f2f93ad44af30d14e31646f4d00615979b2c525394fa373 WatchSource:0}: Error finding container f23c0dd1b15c4d196f2f93ad44af30d14e31646f4d00615979b2c525394fa373: Status 404 returned error can't find the container with id f23c0dd1b15c4d196f2f93ad44af30d14e31646f4d00615979b2c525394fa373
Oct 30 16:22:39.575782 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:39.575753    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:39.575944 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:39.575866    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:22:39.575944 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:39.575922    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls podName:cfe1d12e-6cb7-487f-a352-590c8fcf7775 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.575905728 +0000 UTC m=+30.962784610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls") pod "dns-default-ts8b4" (UID: "cfe1d12e-6cb7-487f-a352-590c8fcf7775") : secret "dns-default-metrics-tls" not found
Oct 30 16:22:40.335810 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:40.335774    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fszqg" event={"ID":"b8853655-ace7-46f7-89f4-8be79930cffa","Type":"ContainerStarted","Data":"f23c0dd1b15c4d196f2f93ad44af30d14e31646f4d00615979b2c525394fa373"}
Oct 30 16:22:40.583227 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:40.583179    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:40.583395 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:40.583328    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:22:40.583457 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:40.583402    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls podName:cfe1d12e-6cb7-487f-a352-590c8fcf7775 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.583385272 +0000 UTC m=+32.970264152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls") pod "dns-default-ts8b4" (UID: "cfe1d12e-6cb7-487f-a352-590c8fcf7775") : secret "dns-default-metrics-tls" not found
Oct 30 16:22:42.599713 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:42.599635    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:42.600068 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:42.599758    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:22:42.600068 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:42.599809    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls podName:cfe1d12e-6cb7-487f-a352-590c8fcf7775 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.599796299 +0000 UTC m=+36.986675179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls") pod "dns-default-ts8b4" (UID: "cfe1d12e-6cb7-487f-a352-590c8fcf7775") : secret "dns-default-metrics-tls" not found
Oct 30 16:22:43.343781 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:43.343746    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fszqg" event={"ID":"b8853655-ace7-46f7-89f4-8be79930cffa","Type":"ContainerStarted","Data":"8ee532b9e8cdc522b7d3354dd8b001718d501e13bf1d2504ee632c59c503e5d7"}
Oct 30 16:22:43.383557 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:43.383505    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fszqg" podStartSLOduration=2.627092347 podStartE2EDuration="5.383491085s" totalImagesPullingTime="2.756398738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:43.383227522 +0000 UTC m=+33.770106421" watchObservedRunningTime="2026-10-30 16:22:43.383491085 +0000 UTC m=+33.770369986"
Oct 30 16:22:43.809010 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:43.808967    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:43.811455 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:43.811435    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:43.881055 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:43.881018    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:43.910071 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:43.910035    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:43.910246 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:43.910081    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:43.910246 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:43.910206    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 16:22:43.910333 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:43.910277    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:23:15.910256802 +0000 UTC m=+66.297135682 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : secret "metrics-daemon-secret" not found
Oct 30 16:22:43.912553 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:43.912528    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:44.185559 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:44.185466    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:44.235812 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:44.235782    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4pp5"]
Oct 30 16:22:44.239451 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:44.239422    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536629a7_e02f_4d49_a9de_cccd0e39132e.slice/crio-b1741e5a58e70e69a43dbc80ceac5b5dc06fbc9e8260e1f78612ee05dcfd5499 WatchSource:0}: Error finding container b1741e5a58e70e69a43dbc80ceac5b5dc06fbc9e8260e1f78612ee05dcfd5499: Status 404 returned error can't find the container with id b1741e5a58e70e69a43dbc80ceac5b5dc06fbc9e8260e1f78612ee05dcfd5499
Oct 30 16:22:44.347335 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:44.347295    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4pp5" event={"ID":"536629a7-e02f-4d49-a9de-cccd0e39132e","Type":"ContainerStarted","Data":"b1741e5a58e70e69a43dbc80ceac5b5dc06fbc9e8260e1f78612ee05dcfd5499"}
Oct 30 16:22:44.515363 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:44.515335    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-htgvf"]
Oct 30 16:22:44.518384 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:44.518359    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea838c0_50c6_4c52_a0ff_e1e019022677.slice/crio-5cae687bc1911979d5a755f7289f247095ac3ac348d4d0717c273b6ab3ef470e WatchSource:0}: Error finding container 5cae687bc1911979d5a755f7289f247095ac3ac348d4d0717c273b6ab3ef470e: Status 404 returned error can't find the container with id 5cae687bc1911979d5a755f7289f247095ac3ac348d4d0717c273b6ab3ef470e
Oct 30 16:22:45.033008 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.032977    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"]
Oct 30 16:22:45.052260 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.052225    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"
Oct 30 16:22:45.058965 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.058933    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"]
Oct 30 16:22:45.067990 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.067967    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 16:22:45.068939 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.068914    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 16:22:45.069058 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.068982    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bls9m\""
Oct 30 16:22:45.118265 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.118228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmn6\" (UniqueName: \"kubernetes.io/projected/ee56683d-41c9-4d01-83c4-57c1c1e184d4-kube-api-access-8bmn6\") pod \"migrator-66df5d496b-pmd76\" (UID: \"ee56683d-41c9-4d01-83c4-57c1c1e184d4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"
Oct 30 16:22:45.218917 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.218882    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmn6\" (UniqueName: \"kubernetes.io/projected/ee56683d-41c9-4d01-83c4-57c1c1e184d4-kube-api-access-8bmn6\") pod \"migrator-66df5d496b-pmd76\" (UID: \"ee56683d-41c9-4d01-83c4-57c1c1e184d4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"
Oct 30 16:22:45.241948 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.241912    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmn6\" (UniqueName: \"kubernetes.io/projected/ee56683d-41c9-4d01-83c4-57c1c1e184d4-kube-api-access-8bmn6\") pod \"migrator-66df5d496b-pmd76\" (UID: \"ee56683d-41c9-4d01-83c4-57c1c1e184d4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"
Oct 30 16:22:45.351828 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.351744    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-htgvf" event={"ID":"bea838c0-50c6-4c52-a0ff-e1e019022677","Type":"ContainerStarted","Data":"5cae687bc1911979d5a755f7289f247095ac3ac348d4d0717c273b6ab3ef470e"}
Oct 30 16:22:45.376848 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.376816    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"
Oct 30 16:22:45.697697 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.697589    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"]
Oct 30 16:22:45.702239 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:45.702203    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee56683d_41c9_4d01_83c4_57c1c1e184d4.slice/crio-288e05c64c1abbdfa232c0da9dbd0d75ffb26fb94c5b1175c9a21885c1f59cfd WatchSource:0}: Error finding container 288e05c64c1abbdfa232c0da9dbd0d75ffb26fb94c5b1175c9a21885c1f59cfd: Status 404 returned error can't find the container with id 288e05c64c1abbdfa232c0da9dbd0d75ffb26fb94c5b1175c9a21885c1f59cfd
Oct 30 16:22:45.897698 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.897656    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6scbv"]
Oct 30 16:22:45.925539 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.925498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:45.926741 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.926710    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6scbv"]
Oct 30 16:22:45.962266 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.961634    2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jj2nm-v6brc"]
Oct 30 16:22:45.975615 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.975581    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:45.985718 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:45.985688    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jj2nm-v6brc"]
Oct 30 16:22:46.026496 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.026460    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lpvkt\""
Oct 30 16:22:46.026723 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.026623    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 16:22:46.026723 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.026697    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 16:22:46.026911 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.026760    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 16:22:46.026911 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.026862    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 16:22:46.027009 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.026942    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 16:22:46.027449 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027427    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.027551 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027486    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmcfr\" (UniqueName: \"kubernetes.io/projected/918257e5-108f-4407-a0a5-1b540606ff05-kube-api-access-bmcfr\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.027551 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027527    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.027653 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.027653 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027595    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/918257e5-108f-4407-a0a5-1b540606ff05-crio-socket\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.027653 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027626    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.027794 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptftn\" (UniqueName: \"kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.027794 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027709    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/918257e5-108f-4407-a0a5-1b540606ff05-data-volume\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.027794 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.027736    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.028205 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.028160    2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6n6gp\""
Oct 30 16:22:46.046770 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.046740    2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 16:22:46.128501 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.128659 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128527    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcfr\" (UniqueName: \"kubernetes.io/projected/918257e5-108f-4407-a0a5-1b540606ff05-kube-api-access-bmcfr\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.128659 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128558    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.128659 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.128659 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128612    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/918257e5-108f-4407-a0a5-1b540606ff05-crio-socket\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.128659 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128636    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.128844 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128667    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptftn\" (UniqueName: \"kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.128844 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128700    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/918257e5-108f-4407-a0a5-1b540606ff05-data-volume\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.128844 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.128725    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.128844 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.128833    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:22:46.128986 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.128903    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.628885139 +0000 UTC m=+37.015764041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : configmap "kube-rbac-proxy" not found
Oct 30 16:22:46.130171 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.130147    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:46.130295 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.130181    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.130295 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.130232    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.630214524 +0000 UTC m=+37.017093407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:46.130691 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.130668    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.130764 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.130743    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/918257e5-108f-4407-a0a5-1b540606ff05-data-volume\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.130862 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.130835    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/918257e5-108f-4407-a0a5-1b540606ff05-crio-socket\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.131031 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.131011    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.155129 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.155096    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcfr\" (UniqueName: \"kubernetes.io/projected/918257e5-108f-4407-a0a5-1b540606ff05-kube-api-access-bmcfr\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.155408 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.155391    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptftn\" (UniqueName: \"kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.311200 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.310900    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:46.362214 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.359578    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76" event={"ID":"ee56683d-41c9-4d01-83c4-57c1c1e184d4","Type":"ContainerStarted","Data":"288e05c64c1abbdfa232c0da9dbd0d75ffb26fb94c5b1175c9a21885c1f59cfd"}
Oct 30 16:22:46.634067 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.633975    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:46.634067 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.634058    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.634302 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.634115    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:46.634302 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.634230    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:22:46.634302 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.634298    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:47.634278503 +0000 UTC m=+38.021157385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : configmap "kube-rbac-proxy" not found
Oct 30 16:22:46.634746 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.634727    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 16:22:46.634805 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.634784    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls podName:cfe1d12e-6cb7-487f-a352-590c8fcf7775 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.634768948 +0000 UTC m=+45.021647831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls") pod "dns-default-ts8b4" (UID: "cfe1d12e-6cb7-487f-a352-590c8fcf7775") : secret "dns-default-metrics-tls" not found
Oct 30 16:22:46.634890 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.634839    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:46.634890 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:46.634871    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:47.634861266 +0000 UTC m=+38.021740149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:46.638406 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:46.638379    2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jj2nm-v6brc"]
Oct 30 16:22:46.649703 ip-10-0-136-74 kubenswrapper[2570]: W1030 16:22:46.649672    2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8aea925_85ca_48f2_a6a3_03eefedf6022.slice/crio-0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e WatchSource:0}: Error finding container 0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e: Status 404 returned error can't find the container with id 0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e
Oct 30 16:22:47.364807 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:47.364771    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jj2nm-v6brc" event={"ID":"c8aea925-85ca-48f2-a6a3-03eefedf6022","Type":"ContainerStarted","Data":"0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e"}
Oct 30 16:22:47.642726 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:47.642634    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:47.642887 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:47.642735    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:47.642957 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:47.642884    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:47.642957 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:47.642947    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:49.642928258 +0000 UTC m=+40.029807154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:47.643070 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:47.642996    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:22:47.643070 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:47.643020    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:49.643012369 +0000 UTC m=+40.029891262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : configmap "kube-rbac-proxy" not found
Oct 30 16:22:48.369145 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:48.369106    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-htgvf" event={"ID":"bea838c0-50c6-4c52-a0ff-e1e019022677","Type":"ContainerStarted","Data":"35cf8971fb33e61a4b828141b83b44c777f53c27b961dbc219a4a46ed45c3cd5"}
Oct 30 16:22:48.369643 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:48.369244    2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:48.370739 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:48.370716    2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76" event={"ID":"ee56683d-41c9-4d01-83c4-57c1c1e184d4","Type":"ContainerStarted","Data":"9e22d357b87ceb10aef0cf5667bdd075cb7724635fb6a689b14b14a5962eec20"}
Oct 30 16:22:48.408543 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:48.408476    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-htgvf" podStartSLOduration=34.706466025 podStartE2EDuration="38.408456798s" totalImagesPullingTime="3.701990773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:48.407518194 +0000 UTC m=+38.794397097" watchObservedRunningTime="2026-10-30 16:22:48.408456798 +0000 UTC m=+38.795335701"
Oct 30 16:22:48.446491 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:48.446442    2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76" podStartSLOduration=1.759081282 podStartE2EDuration="3.446428245s" totalImagesPullingTime="1.687346963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:48.446388202 +0000 UTC m=+38.833267103" watchObservedRunningTime="2026-10-30 16:22:48.446428245 +0000 UTC m=+38.833307147"
Oct 30 16:22:49.657755 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:49.657717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:49.658222 ip-10-0-136-74 kubenswrapper[2570]: I1030 16:22:49.657786    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:49.658222 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:49.657861    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 16:22:49.658222 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:49.657889    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:49.658222 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:49.657938    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:53.657924771 +0000 UTC m=+44.044803664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:49.658222 ip-10-0-136-74 kubenswrapper[2570]: E1030 16:22:49.657952    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:53.657946165 +0000 UTC m=+44.044825044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : configmap "kube-rbac-proxy" not found
Oct 30 16:22:50.079607 ip-10-0-136-74 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 16:22:50.095612 ip-10-0-136-74 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 16:22:50.095787 ip-10-0-136-74 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 16:22:50.095858 ip-10-0-136-74 systemd[1]: kubelet.service: Consumed 2.396s CPU time.
Oct 30 16:22:50.107871 ip-10-0-136-74 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 16:22:50.217906 ip-10-0-136-74 kubenswrapper[4497]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:22:50.217906 ip-10-0-136-74 kubenswrapper[4497]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 16:22:50.217906 ip-10-0-136-74 kubenswrapper[4497]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:22:50.217906 ip-10-0-136-74 kubenswrapper[4497]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 16:22:50.220017 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.219934    4497 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:50.220017 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220016    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:50.220017 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220020    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220024    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220027    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220030    4497 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220033    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220036    4497 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220039    4497 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220042    4497 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220045    4497 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220048    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220050    4497 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220059    4497 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220078    4497 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220083    4497 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220086    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220089    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220092    4497 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220095    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220098    4497 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220101    4497 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:50.220123 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220104    4497 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220107    4497 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220110    4497 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220113    4497 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220116    4497 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220119    4497 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220122    4497 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220125    4497 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220128    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220131    4497 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220134    4497 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220137    4497 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220139    4497 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220143    4497 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220148    4497 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220152    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220154    4497 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220158    4497 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220162    4497 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:50.220633 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220164    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220169    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220172    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220174    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220177    4497 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220180    4497 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220197    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220201    4497 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220204    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220208    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220211    4497 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220214    4497 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220217    4497 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220220    4497 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220222    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220226    4497 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220229    4497 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220231    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220234    4497 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:50.221103 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220237    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220240    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220242    4497 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220245    4497 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220248    4497 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220251    4497 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220253    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220256    4497 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220259    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220261    4497 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220264    4497 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220266    4497 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220269    4497 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220272    4497 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220275    4497 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220278    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220281    4497 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220284    4497 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220287    4497 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220289    4497 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:50.221659 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220292    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220294    4497 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220298    4497 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220301    4497 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220303    4497 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220306    4497 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220309    4497 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220311    4497 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220314    4497 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220317    4497 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220320    4497 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220322    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220325    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220328    4497 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220330    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220333    4497 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220336    4497 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220338    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220341    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:50.222199 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220343    4497 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220346    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220348    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220351    4497 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220354    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220357    4497 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.220360    4497 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221170    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221178    4497 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221201    4497 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221208    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221212    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221215    4497 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221219    4497 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221222    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221226    4497 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221229    4497 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221232    4497 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221235    4497 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:50.222703 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221237    4497 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221240    4497 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221244    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221246    4497 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221249    4497 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221251    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221254    4497 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221257    4497 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221259    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221262    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221265    4497 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221267    4497 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221270    4497 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221273    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221275    4497 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221278    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221280    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221283    4497 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221285    4497 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221288    4497 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:50.223168 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221290    4497 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221293    4497 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221295    4497 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221298    4497 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221301    4497 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221304    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221306    4497 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221309    4497 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221312    4497 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221315    4497 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221317    4497 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221320    4497 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221323    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221325    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221329    4497 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221331    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221334    4497 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221337    4497 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221339    4497 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:50.223702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221342    4497 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221345    4497 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221347    4497 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221350    4497 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221353    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221355    4497 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221358    4497 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221360    4497 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221363    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221366    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221369    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221371    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221373    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221376    4497 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221379    4497 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221382    4497 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221384    4497 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221388    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221392    4497 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221396    4497 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:50.224182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221399    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221402    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221405    4497 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221408    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221411    4497 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221414    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221417    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221420    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221422    4497 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221425    4497 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221429    4497 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221433    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221436    4497 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221440    4497 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221443    4497 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221446    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221449    4497 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221453    4497 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221456    4497 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:50.224670 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221458    4497 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221461    4497 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221464    4497 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221466    4497 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221469    4497 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221471    4497 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221474    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221476    4497 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221479    4497 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221481    4497 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221484    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221487    4497 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221490    4497 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221492    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221495    4497 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.221498    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222091    4497 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222100    4497 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222102    4497 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222105    4497 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:50.225121 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222108    4497 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222111    4497 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222114    4497 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222117    4497 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222119    4497 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222122    4497 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222124    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222127    4497 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222130    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222132    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222135    4497 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222138    4497 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222140    4497 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222146    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222154    4497 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222157    4497 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222160    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222163    4497 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222166    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222169    4497 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:50.225719 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222171    4497 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222174    4497 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222177    4497 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222199    4497 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222203    4497 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222207    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222210    4497 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222213    4497 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222216    4497 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222219    4497 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222221    4497 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222224    4497 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222226    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222229    4497 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222232    4497 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222234    4497 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222237    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222239    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222242    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222244    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:50.226209 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222247    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222250    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222252    4497 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222255    4497 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222257    4497 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222262    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222265    4497 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222267    4497 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222270    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222272    4497 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222275    4497 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222278    4497 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222280    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222283    4497 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222286    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222288    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222297    4497 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222300    4497 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222302    4497 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222305    4497 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:50.226711 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222308    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222310    4497 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222313    4497 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222315    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222318    4497 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222320    4497 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222323    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222326    4497 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222328    4497 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222331    4497 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222333    4497 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222336    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222338    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222341    4497 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222343    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222346    4497 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222348    4497 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222352    4497 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222355    4497 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:50.227208 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222378    4497 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222383    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222387    4497 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222391    4497 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222395    4497 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222398    4497 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222401    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222404    4497 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222407    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222410    4497 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222419    4497 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222423    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222426    4497 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222429    4497 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222432    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222435    4497 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222438    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222440    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222443    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:50.227662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222445    4497 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222448    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222450    4497 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.222453    4497 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222551    4497 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222559    4497 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222569    4497 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222574    4497 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222581    4497 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222586    4497 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222593    4497 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222599    4497 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222602    4497 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222606    4497 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222609    4497 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222613    4497 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222616    4497 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222619    4497 flags.go:64] FLAG: --cgroup-root=""
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222622    4497 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222625    4497 flags.go:64] FLAG: --client-ca-file=""
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222628    4497 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222631    4497 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222639    4497 flags.go:64] FLAG: --cluster-domain=""
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222642    4497 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 16:22:50.228291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222649    4497 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222652    4497 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222656    4497 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222661    4497 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222664    4497 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222667    4497 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222670    4497 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222674    4497 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222677    4497 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222680    4497 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222683    4497 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222686    4497 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222691    4497 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222694    4497 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222697    4497 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222700    4497 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222703    4497 flags.go:64] FLAG: --enable-server="true"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222706    4497 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222714    4497 flags.go:64] FLAG: --event-burst="100"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222718    4497 flags.go:64] FLAG: --event-qps="50"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222722    4497 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222725    4497 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222728    4497 flags.go:64] FLAG: --eviction-hard=""
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222733    4497 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 16:22:50.228970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222736    4497 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222739    4497 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222742    4497 flags.go:64] FLAG: --eviction-soft=""
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222745    4497 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222748    4497 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222751    4497 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222755    4497 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222758    4497 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222762    4497 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222766    4497 flags.go:64] FLAG: --feature-gates=""
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222770    4497 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222773    4497 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222777    4497 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222780    4497 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222783    4497 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222787    4497 flags.go:64] FLAG: --help="false"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222790    4497 flags.go:64] FLAG: --hostname-override="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222793    4497 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222796    4497 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222799    4497 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222802    4497 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222806    4497 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222809    4497 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 16:22:50.229646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222812    4497 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222815    4497 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222818    4497 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222821    4497 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222825    4497 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222829    4497 flags.go:64] FLAG: --kube-reserved=""
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222832    4497 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222835    4497 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222838    4497 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222841    4497 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222843    4497 flags.go:64] FLAG: --lock-file=""
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222846    4497 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222850    4497 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222853    4497 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222858    4497 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222861    4497 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222865    4497 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222869    4497 flags.go:64] FLAG: --logging-format="text"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222872    4497 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222876    4497 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222879    4497 flags.go:64] FLAG: --manifest-url=""
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222882    4497 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222887    4497 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222890    4497 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222894    4497 flags.go:64] FLAG: --max-pods="110"
Oct 30 16:22:50.230401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222897    4497 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222900    4497 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222903    4497 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222906    4497 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222909    4497 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222912    4497 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222915    4497 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222922    4497 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222926    4497 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222929    4497 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222932    4497 flags.go:64] FLAG: --pod-cidr=""
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222936    4497 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222939    4497 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222942    4497 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222945    4497 flags.go:64] FLAG: --port="10250"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222948    4497 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222951    4497 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0886c9cb95e15c7f3"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222955    4497 flags.go:64] FLAG: --qos-reserved=""
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222957    4497 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222960    4497 flags.go:64] FLAG: --register-node="true"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222963    4497 flags.go:64] FLAG: --register-with-taints=""
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222967    4497 flags.go:64] FLAG: --registry-burst="10"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222970    4497 flags.go:64] FLAG: --registry-qps="5"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222973    4497 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222977    4497 flags.go:64] FLAG: --reserved-memory=""
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222981    4497 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 16:22:50.231323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222985    4497 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222988    4497 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222991    4497 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222994    4497 flags.go:64] FLAG: --runonce="false"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.222997    4497 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223000    4497 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223004    4497 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223007    4497 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223009    4497 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223013    4497 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223016    4497 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223019    4497 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223022    4497 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223025    4497 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223028    4497 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223031    4497 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223034    4497 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223038    4497 flags.go:64] FLAG: --system-cgroups=""
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223041    4497 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223046    4497 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223049    4497 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223053    4497 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223058    4497 flags.go:64] FLAG: --tls-min-version=""
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223061    4497 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223064    4497 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 16:22:50.231990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223067    4497 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223070    4497 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223073    4497 flags.go:64] FLAG: --v="2"
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223077    4497 flags.go:64] FLAG: --version="false"
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223081    4497 flags.go:64] FLAG: --vmodule=""
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223087    4497 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223090    4497 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223257    4497 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223262    4497 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223266    4497 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223269    4497 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223272    4497 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223274    4497 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223277    4497 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223280    4497 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223283    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223286    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223288    4497 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223291    4497 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223295    4497 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223299    4497 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:50.232604 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223303    4497 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223306    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223309    4497 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223313    4497 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223315    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223318    4497 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223321    4497 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223323    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223326    4497 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223329    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223331    4497 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223334    4497 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223337    4497 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223339    4497 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223343    4497 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223346    4497 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223351    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223354    4497 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223357    4497 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:50.233102 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223360    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223362    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223365    4497 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223368    4497 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223371    4497 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223373    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223376    4497 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223378    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223381    4497 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223384    4497 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223386    4497 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223389    4497 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223391    4497 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223394    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223397    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223399    4497 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223403    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223406    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223409    4497 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:50.233587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223411    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223414    4497 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223417    4497 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223419    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223422    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223425    4497 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223428    4497 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223430    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223433    4497 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223435    4497 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223439    4497 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223442    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223445    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223447    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223450    4497 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223452    4497 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223456    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223459    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223462    4497 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223465    4497 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:50.234158 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223467    4497 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223471    4497 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223475    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223479    4497 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223482    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223486    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223491    4497 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223495    4497 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223499    4497 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223505    4497 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223510    4497 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223514    4497 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223518    4497 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223522    4497 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223526    4497 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223530    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223535    4497 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223540    4497 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223544    4497 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223548    4497 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:50.234985 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223552    4497 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223556    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223563    4497 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223566    4497 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223569    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223572    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223575    4497 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223577    4497 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223580    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223584    4497 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223587    4497 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223590    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223593    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223595    4497 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.223631    4497 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223684    4497 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:50.235645 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223692    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223698    4497 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223702    4497 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223706    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223711    4497 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223715    4497 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223723    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223727    4497 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223731    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223735    4497 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223740    4497 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223744    4497 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223748    4497 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223754    4497 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223760    4497 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223764    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223768    4497 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223774    4497 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223779    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223783    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:50.236148 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223788    4497 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223792    4497 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223797    4497 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223801    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223805    4497 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223810    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223815    4497 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223819    4497 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223823    4497 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223827    4497 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223832    4497 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223836    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223840    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223844    4497 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223848    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223853    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223857    4497 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223861    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223867    4497 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:50.236895 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223871    4497 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223875    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223879    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223884    4497 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223888    4497 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223892    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223896    4497 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223898    4497 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223901    4497 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223904    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223907    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223909    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223912    4497 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223915    4497 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223918    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223920    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223923    4497 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223926    4497 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223929    4497 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223932    4497 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:50.237556 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223936    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223938    4497 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223941    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223944    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223948    4497 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223950    4497 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223953    4497 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223955    4497 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223958    4497 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223960    4497 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223963    4497 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223967    4497 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223970    4497 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223973    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223976    4497 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223978    4497 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223981    4497 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223984    4497 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223987    4497 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223989    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:50.238201 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223992    4497 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223995    4497 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.223997    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224001    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224004    4497 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224007    4497 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224009    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224012    4497 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224015    4497 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224019    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224021    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224024    4497 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224027    4497 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224030    4497 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224033    4497 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224036    4497 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224038    4497 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224041    4497 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224044    4497 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224046    4497 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:50.238927 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224049    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224052    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224054    4497 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224061    4497 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224066    4497 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.224070    4497 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.227944    4497 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.227961    4497 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.227979    4497 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.227984    4497 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228042    4497 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228048    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228051    4497 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228054    4497 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228059    4497 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228062    4497 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228065    4497 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228068    4497 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228071    4497 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:50.239517 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228073    4497 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228076    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228079    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228082    4497 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228085    4497 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228089    4497 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228092    4497 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228096    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228098    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228101    4497 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228104    4497 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228107    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228109    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228112    4497 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228115    4497 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228117    4497 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228120    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228123    4497 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228126    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228129    4497 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:50.239997 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228132    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228135    4497 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228138    4497 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228140    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228143    4497 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228146    4497 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228149    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228153    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228156    4497 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228159    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228161    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228164    4497 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228168    4497 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228172    4497 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228176    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228178    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228181    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228203    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:50.240691 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228206    4497 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228210    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228213    4497 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228216    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228218    4497 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228221    4497 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228224    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228226    4497 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228229    4497 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228231    4497 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228234    4497 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228236    4497 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228239    4497 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228242    4497 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228245    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228248    4497 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228250    4497 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228254    4497 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228256    4497 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228259    4497 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:50.241132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228262    4497 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228264    4497 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228267    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228270    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228272    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228275    4497 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228278    4497 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228280    4497 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228283    4497 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228286    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228288    4497 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228291    4497 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228294    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228296    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228299    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228302    4497 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228305    4497 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228308    4497 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228310    4497 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228313    4497 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:50.241639 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228316    4497 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228319    4497 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228322    4497 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228325    4497 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228327    4497 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228330    4497 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228332    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228335    4497 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228338    4497 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228340    4497 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228343    4497 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228346    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228348    4497 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228351    4497 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228354    4497 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228356    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228359    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228362    4497 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:50.242143 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228364    4497 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.228393    4497 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228549    4497 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228555    4497 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228558    4497 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228561    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228564    4497 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228566    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228569    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228573    4497 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228575    4497 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228578    4497 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228581    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228583    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228586    4497 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228589    4497 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:50.242598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228591    4497 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228595    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228598    4497 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228600    4497 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228603    4497 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228605    4497 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228608    4497 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228610    4497 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228613    4497 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228615    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228618    4497 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228620    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228623    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228626    4497 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228629    4497 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228631    4497 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228634    4497 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228636    4497 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228639    4497 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:50.242982 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228641    4497 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228644    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228647    4497 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228649    4497 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228652    4497 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228654    4497 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228657    4497 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228660    4497 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228662    4497 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228665    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228668    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228670    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228673    4497 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228675    4497 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228678    4497 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228681    4497 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228683    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228686    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228688    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228691    4497 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:50.243605 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228694    4497 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228696    4497 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228699    4497 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228701    4497 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228704    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228709    4497 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228712    4497 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228714    4497 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228717    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228719    4497 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228722    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228724    4497 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228727    4497 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228730    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228732    4497 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228735    4497 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228738    4497 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228740    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228743    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228747    4497 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:50.244097 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228752    4497 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228755    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228758    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228761    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228763    4497 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228766    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228769    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228772    4497 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228774    4497 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228777    4497 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228780    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228782    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228785    4497 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228788    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228790    4497 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228793    4497 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228795    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228800    4497 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228803    4497 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:50.244593 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228806    4497 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228808    4497 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228811    4497 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228813    4497 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228816    4497 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228818    4497 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228823    4497 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228827    4497 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228830    4497 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228833    4497 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228836    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228838    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228841    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228843    4497 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.228867    4497 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228900    4497 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 16:22:50.245076 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228905    4497 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228908    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228912    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228915    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228917    4497 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228920    4497 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228923    4497 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228925    4497 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228928    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228931    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228933    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228936    4497 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228939    4497 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228941    4497 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228946    4497 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228948    4497 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228951    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228954    4497 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228957    4497 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228959    4497 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 16:22:50.245473 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228962    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228964    4497 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228967    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228969    4497 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228972    4497 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228975    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228978    4497 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228981    4497 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228983    4497 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228986    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228989    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228993    4497 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.228997    4497 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229000    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229003    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229006    4497 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229008    4497 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229011    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229014    4497 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 16:22:50.245952 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229016    4497 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229019    4497 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229021    4497 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229024    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229026    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229029    4497 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229033    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229036    4497 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229039    4497 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229041    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229044    4497 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229047    4497 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229049    4497 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229052    4497 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229054    4497 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229057    4497 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229059    4497 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229063    4497 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229067    4497 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 16:22:50.246449 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229070    4497 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229073    4497 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229076    4497 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229079    4497 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229081    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229084    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229087    4497 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229090    4497 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229093    4497 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229095    4497 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229098    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229100    4497 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229103    4497 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229105    4497 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229108    4497 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229111    4497 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229113    4497 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229115    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229118    4497 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229121    4497 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 16:22:50.246919 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229124    4497 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229127    4497 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229130    4497 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229134    4497 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229137    4497 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229139    4497 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229142    4497 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229145    4497 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229147    4497 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229150    4497 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229153    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229155    4497 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229158    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229160    4497 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229163    4497 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229166    4497 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229168    4497 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229171    4497 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229174    4497 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 16:22:50.247724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229177    4497 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229179    4497 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229195    4497 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229200    4497 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229205    4497 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229209    4497 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229212    4497 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:50.229216    4497 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.229271    4497 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.231364    4497 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.231434    4497 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.231662    4497 server.go:1014] "Starting client certificate rotation"
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.231798    4497 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.231849    4497 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:17:10 +0000 UTC" deadline="2028-04-21 03:00:56.759619666 +0000 UTC"
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.231876    4497 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12922h38m6.527745327s"
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.232107    4497 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:22:50.248587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.233216    4497 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 16:22:50.249098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.234850    4497 log.go:25] "Validated CRI v1 runtime API"
Oct 30 16:22:50.249098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.236558    4497 log.go:25] "Validated CRI v1 image API"
Oct 30 16:22:50.249098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.237362    4497 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 16:22:50.249098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.239013    4497 fs.go:135] Filesystem UUIDs: map[0e86a780-c38a-4fe9-9c56-21c7506d9f2a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8de81c42-5d50-4c38-a09e-cd4bbdec3d79:/dev/nvme0n1p4]
Oct 30 16:22:50.249268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.239040    4497 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b0dc445a7833cd606d2b04a1f8258674bb82b15dc1fabf3bf6e65de59621145/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b0dc445a7833cd606d2b04a1f8258674bb82b15dc1fabf3bf6e65de59621145/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11df67c1bef2bea44c38376b183f4ae806638d276b0cb84089db06854c152b23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11df67c1bef2bea44c38376b183f4ae806638d276b0cb84089db06854c152b23/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d10232299381461a1bf6b7e96c8cdce29e34cd1a0cc1d1638e92f602acea66a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d10232299381461a1bf6b7e96c8cdce29e34cd1a0cc1d1638e92f602acea66a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/288e05c64c1abbdfa232c0da9dbd0d75ffb26fb94c5b1175c9a21885c1f59cfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/288e05c64c1abbdfa232c0da9dbd0d75ffb26fb94c5b1175c9a21885c1f59cfd/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b94dc792042463d6e9ee676b30c0d573c34fa773a130179072cfaa6230e158f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b94dc792042463d6e9ee676b30c0d573c34fa773a130179072cfaa6230e158f/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cae687bc1911979d5a755f7289f247095ac3ac348d4d0717c273b6ab3ef470e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cae687bc1911979d5a755f7289f247095ac3ac348d4d0717c273b6ab3ef470e/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/694805d01bf18c4c078132aea8a4e991a4f2e6edede8e355c9adb01cf2b7b983/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/694805d01bf18c4c078132aea8a4e991a4f2e6edede8e355c9adb01cf2b7b983/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fa79b6ad1e3fd8ceac59474a0e16df925ed4b3beb27b4ef5f7b192bb793cfbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fa79b6ad1e3fd8ceac59474a0e16df925ed4b3beb27b4ef5f7b192bb793cfbc/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ab6d0bd18409d2c903badb6074cca41ba8bcf67965b2f5cdca4078b5ec9ad34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ab6d0bd18409d2c903badb6074cca41ba8bcf67965b2f5cdca4078b5ec9ad34/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85571300fa7ff721d661ab49ad448ae4f261d35b1cb114eaa19e194e6ad60388/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85571300fa7ff721d661ab49ad448ae4f261d35b1cb114eaa19e194e6ad60388/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b13f2f43a31ccadae64086dd32f85505f60d7a370b57b1f32042ca27f351758b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b13f2f43a31ccadae64086dd32f85505f60d7a370b57b1f32042ca27f351758b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1741e5a58e70e69a43dbc80ceac5b5dc06fbc9e8260e1f78612ee05dcfd5499/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1741e5a58e70e69a43dbc80ceac5b5dc06fbc9e8260e1f78612ee05dcfd5499/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec42e889039300d4ce8157ac0177cafa75e1397bb94c07394e66beaedbc228dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec42e889039300d4ce8157ac0177cafa75e1397bb94c07394e66beaedbc228dd/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f23c0dd1b15c4d196f2f93ad44af30d14e31646f4d00615979b2c525394fa373/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f23c0dd1b15c4d196f2f93ad44af30d14e31646f4d00615979b2c525394fa373/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8/volumes/kubernetes.io~projected/kube-api-access-qmfr2:{mountpoint:/var/lib/kubelet/pods/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8/volumes/kubernetes.io~projected/kube-api-access-qmfr2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~projected/kube-api-access-vkglq:{mountpoint:/var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~projected/kube-api-access-vkglq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a046e81-b1ab-4da2-ac50-7d06f2c30668/volumes/kubernetes.io~projected/kube-api-access-2xmb7:{mountpoint:/var/lib/kubelet/pods/3a046e81-b1ab-4da2-ac50-7d06f2c30668/volumes/kubernetes.io~projected/kube-api-access-2xmb7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c365930-eba1-4476-8a6c-7efe851313a8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3c365930-eba1-4476-8a6c-7efe851313a8/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/529ebf7f-7783-4575-a1fd-5c672e410f6a/volumes/kubernetes.io~projected/kube-api-access-9kctn:{mountpoint:/var/lib/kubelet/pods/529ebf7f-7783-4575-a1fd-5c672e410f6a/volumes/kubernetes.io~projected/kube-api-access-9kctn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/536629a7-e02f-4d49-a9de-cccd0e39132e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/536629a7-e02f-4d49-a9de-cccd0e39132e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57050003-649e-4fad-ba51-9f1a534a859c/volumes/kubernetes.io~projected/kube-api-access-9p46b:{mountpoint:/var/lib/kubelet/pods/57050003-649e-4fad-ba51-9f1a534a859c/volumes/kubernetes.io~projected/kube-api-access-9p46b major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volumes/kubernetes.io~projected/kube-api-access-wc4gx:{mountpoint:/var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volumes/kubernetes.io~projected/kube-api-access-wc4gx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b89a9fa-bf4d-44f6-9331-22a216a06f6d/volumes/kubernetes.io~projected/kube-api-access-jdvbj:{mountpoint:/var/lib/kubelet/pods/8b89a9fa-bf4d-44f6-9331-22a216a06f6d/volumes/kubernetes.io~projected/kube-api-access-jdvbj major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/918257e5-108f-4407-a0a5-1b540606ff05/volumes/kubernetes.io~projected/kube-api-access-bmcfr:{mountpoint:/var/lib/kubelet/pods/918257e5-108f-4407-a0a5-1b540606ff05/volumes/kubernetes.io~projected/kube-api-access-bmcfr major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6d1b009-1219-4dba-97a5-32d9f88b1447/volumes/kubernetes.io~projected/kube-api-access-ppj6l:{mountpoint:/var/lib/kubelet/pods/b6d1b009-1219-4dba-97a5-32d9f88b1447/volumes/kubernetes.io~projected/kube-api-access-ppj6l major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8853655-ace7-46f7-89f4-8be79930cffa/volumes/kubernetes.io~projected/kube-api-access-n2tjs:{mountpoint:/var/lib/kubelet/pods/b8853655-ace7-46f7-89f4-8be79930cffa/volumes/kubernetes.io~projected/kube-api-access-n2tjs major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8853655-ace7-46f7-89f4-8be79930cffa/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b8853655-ace7-46f7-89f4-8be79930cffa/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b88993de-a05b-40ff-a753-8b9949f5f6c4/volumes/kubernetes.io~projected/kube-api-access-s2d2w:{mountpoint:/var/lib/kubelet/pods/b88993de-a05b-40ff-a753-8b9949f5f6c4/volumes/kubernetes.io~projected/kube-api-access-s2d2w major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bea838c0-50c6-4c52-a0ff-e1e019022677/volumes/kubernetes.io~projected/kube-api-access-4qgv9:{mountpoint:/var/lib/kubelet/pods/bea838c0-50c6-4c52-a0ff-e1e019022677/volumes/kubernetes.io~projected/kube-api-access-4qgv9 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8aea925-85ca-48f2-a6a3-03eefedf6022/volumes/kubernetes.io~projected/kube-api-access-ptftn:{mountpoint:/var/lib/kubelet/pods/c8aea925-85ca-48f2-a6a3-03eefedf6022/volumes/kubernetes.io~projected/kube-api-access-ptftn major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfe1d12e-6cb7-487f-a352-590c8fcf7775/volumes/kubernetes.io~projected/kube-api-access-m46dw:{mountpoint:/var/lib/kubelet/pods/cfe1d12e-6cb7-487f-a352-590c8fcf7775/volumes/kubernetes.io~projected/kube-api-access-m46dw major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee56683d-41c9-4d01-83c4-57c1c1e184d4/volumes/kubernetes.io~projected/kube-api-access-8bmn6:{mountpoint:/var/lib/kubelet/pods/ee56683d-41c9-4d01-83c4-57c1c1e184d4/volumes/kubernetes.io~projected/kube-api-access-8bmn6 major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2f98e08b27f75130f4d9bfb8e1a18fbf7526cdc2f27add94a509663b2414350d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fd68d94d532f1eef388b2eeed8e8e35399d5f739bbebcf774d2bd3a8dbbac88d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ccdf17cccf9fa54fd94fbb7ffbfb079119fa90098775d9ea05ad04053b8d8baf/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1abafe4abc9fc301f58c7ac5e398b14c6f95596082a6369e82b2f94a629a068e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/76595d936568aeff97ec798c04e846575471ec5abbe3cc77476294af92c02d9b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7a4c21e5f235fbecd0fd6602f5a776551578af1017cb510a1c74b5d0ce2547e9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5d757f2bec459d6f732fb19bdc64a8f861b5d807afe695c0526f72c5125f1dd2/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ed4e16b271c4af496648e2de0486c23feb876add81362d34b870cebf02d81035/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8d561893c2e4e8019a490edb062d7ff5997c76fde1d4cf91a0193d35b047d70d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0f6be27b6b0ac293c4fc80fe02405abcf35699412c6ee87ad101beb72709b9f1/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/8c23ad97edb9b7afde398f547ca365eba5322ec836303859abf6a262c40be50c/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/abc145fb8dd706c770cc69bf64f5c24135fcb2b3d0c3748f41d211de8aa0aae0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/5fee79cc99b7649eb2d15601daa9250d0448135173b7322274cf0d793f92b66a/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/067dc60b9f61c11c5040fce012687630942d1b5a5e4f494f5f21f2f37642d30b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/99d7f5222366f9e8e39b237499ec3857f15539466433dff470fe4cf4b8b82277/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f775edf109fb1db1933879c3c708f958b26617036a727c071c1a72858f1e6448/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bcc6c3a59f2f8bb4970d116a64933c552682da73dcba0f711fbf0d4874602543/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c5cd63b904e97cc585617822326032876f33c02c03cd79562d885479a3d28bf6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e98e41f38509b739048e672124dc587c334c689ae21f39c24b591d994d3a3b95/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/dccf88b46bc10da0292f59939f3f67c45474c3b0477a7dbe5517464b2bda3eef/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/eb7a20831d2d7b57ad33df953626a92e46e620f5029ba17b600d704a29f77fef/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/565349d44fedc5817fc076c067b3e2a9311054ccd0fe26ac2c19b8fe7b6a7a96/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/4bd9a469c57d1a63be970e46a8bc3c9042d8f7b9cf50713a058a8f5a7da20aec/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/15385450b0b7c051ae8ceb629d068a6ed36f0e37b618f8f75d27f6f36280e4e1/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/7899feb83103f2583fd5ecc3db05227b05aff1bf71c73350cd9f619a0c31ea30/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/c8214baffdcbdabff179bf7ecd3ffb2857c1eaa1ccd1f87f08b6b2df587be1bc/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/66a71edc23e2d9e3068b8b657b5b6012af42601807d70d317f054f48ab7ee43c/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/e21ed1fb97f8826d9371fa6e4318bf82b47415b95e1a43f6a32b2d8fc06106e6/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/54ba01dd9e1c52f380a706e5d18cd6f78328a6c8d876e146be7fe8d6c4a0b0fb/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/1f4b89689f16e9c35ee638c801c2ee6f047a97a5592cc152f4d40c0be2f47ae5/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/bcbabf9e84d80e2891a6550e978e9f56a16e47784ac5e601493f041fd51562b7/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/cb094f52475c3b221d01a353d1a7e5b4ea6ea039b7d2a2678c0d80a6b42d195d/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7d0b33128472dc5590687865fb2026cc93265cc048df1d3a218ce6c24ba0bd12/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/abb425f589c2c0c96fafb3acc86fa18c47a231bc55d073c03fcffae944524976/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/e8cfa8eef49f810e8d8c99c0afa26e369f88e5c29d5a11c8748f65a777c5c1a7/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/cb151fbe0cdbe2d64580d698ceb6784ea2842fed9f029cdccca415a27e7df714/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0e2b6110ccacaa2cc5ce32eec6f63082d2391cf01554a6dc81aa8666d0d5e8e5/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/4fcb5362cf9e57a10a92659c7f39cfc125f772335355383e8a9c1b8c2f896c82/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/cd18eea459dee3ee5327d6458f8cfbe1d98594f2219185b79961c57b245b22a5/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d2b9807fb2dca27ba813d6abd740c76e199afb148bada2db4d23177ad3f80b13/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/58ac8d671b90d95e7a81aa73b8b2e3cdf57536877c4d0410cb6334c9c92afa26/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.245350    4497 manager.go:217] Machine: {Timestamp:2026-10-30 16:22:50.244875336 +0000 UTC m=+0.098753685 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100076 MemoryCapacity:33164345344 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2095f3ba0ace843d346f74763c0694 SystemUUID:ec2095f3-ba0a-ce84-3d34-6f74763c0694 BootID:3119779a-a1d7-4a50-be80-ded1a93977fb Filesystems:[{Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fa79b6ad1e3fd8ceac59474a0e16df925ed4b3beb27b4ef5f7b192bb793cfbc/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/694805d01bf18c4c078132aea8a4e991a4f2e6edede8e355c9adb01cf2b7b983/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volumes/kubernetes.io~projected/kube-api-access-wc4gx DeviceMajor:0 DeviceMinor:67 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec42e889039300d4ce8157ac0177cafa75e1397bb94c07394e66beaedbc228dd/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/3a046e81-b1ab-4da2-ac50-7d06f2c30668/volumes/kubernetes.io~projected/kube-api-access-2xmb7 DeviceMajor:0 DeviceMinor:61 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b94dc792042463d6e9ee676b30c0d573c34fa773a130179072cfaa6230e158f/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d10232299381461a1bf6b7e96c8cdce29e34cd1a0cc1d1638e92f602acea66a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~projected/kube-api-access-vkglq DeviceMajor:0 DeviceMinor:60 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ab6d0bd18409d2c903badb6074cca41ba8bcf67965b2f5cdca4078b5ec9ad34/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b0dc445a7833cd606d2b04a1f8258674bb82b15dc1fabf3bf6e65de59621145/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/b88993de-a05b-40ff-a753-8b9949f5f6c4/volumes/kubernetes.io~projected/kube-api-access-s2d2w DeviceMajor:0 DeviceMinor:64 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8/volumes/kubernetes.io~projected/kube-api-access-qmfr2 DeviceMajor:0 DeviceMinor:65 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8aea925-85ca-48f2-a6a3-03eefedf6022/volumes/kubernetes.io~projected/kube-api-access-ptftn DeviceMajor:0 DeviceMinor:228 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57050003-649e-4fad-ba51-9f1a534a859c/volumes/kubernetes.io~projected/kube-api-access-9p46b DeviceMajor:0 DeviceMinor:62 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11df67c1bef2bea44c38376b183f4ae806638d276b0cb84089db06854c152b23/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/b8853655-ace7-46f7-89f4-8be79930cffa/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/536629a7-e02f-4d49-a9de-cccd0e39132e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8853655-ace7-46f7-89f4-8be79930cffa/volumes/kubernetes.io~projected/kube-api-access-n2tjs DeviceMajor:0 DeviceMinor:197 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c365930-eba1-4476-8a6c-7efe851313a8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85571300fa7ff721d661ab49ad448ae4f261d35b1cb114eaa19e194e6ad60388/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b13f2f43a31ccadae64086dd32f85505f60d7a370b57b1f32042ca27f351758b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b6d1b009-1219-4dba-97a5-32d9f88b1447/volumes/kubernetes.io~projected/kube-api-access-ppj6l DeviceMajor:0 DeviceMinor:59 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/8b89a9fa-bf4d-44f6-9331-22a216a06f6d/volumes/kubernetes.io~projected/kube-api-access-jdvbj DeviceMajor:0 DeviceMinor:63 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f23c0dd1b15c4d196f2f93ad44af30d14e31646f4d00615979b2c525394fa373/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/31bd08bb-58ea-4508-8e99-f4a1d9b552ad/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bea838c0-50c6-4c52-a0ff-e1e019022677/volumes/kubernetes.io~projected/kube-api-access-4qgv9 DeviceMajor:0 DeviceMinor:213 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfe1d12e-6cb7-487f-a352-590c8fcf7775/volumes/kubernetes.io~projected/kube-api-access-m46dw DeviceMajor:0 DeviceMinor:201 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/288e05c64c1abbdfa232c0da9dbd0d75ffb26fb94c5b1175c9a21885c1f59cfd/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cae687bc1911979d5a755f7289f247095ac3ac348d4d0717c273b6ab3ef470e/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/529ebf7f-7783-4575-a1fd-5c672e410f6a/volumes/kubernetes.io~projected/kube-api-access-9kctn DeviceMajor:0 DeviceMinor:66 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee56683d-41c9-4d01-83c4-57c1c1e184d4/volumes/kubernetes.io~projected/kube-api-access-8bmn6 DeviceMajor:0 DeviceMinor:222 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83bc0294-07c9-4087-88cf-1cf891d84118/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/918257e5-108f-4407-a0a5-1b540606ff05/volumes/kubernetes.io~projected/kube-api-access-bmcfr DeviceMajor:0 DeviceMinor:227 Capacity:31985745920 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1741e5a58e70e69a43dbc80ceac5b5dc06fbc9e8260e1f78612ee05dcfd5499/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0cc77813161a10e MacAddress:56:dc:be:0c:7f:f5 Speed:10000 Mtu:8901} {Name:288e05c64c1abbd MacAddress:0e:79:f2:e9:3a:a7 Speed:10000 Mtu:8901} {Name:5cae687bc191197 MacAddress:8a:a1:68:fa:37:32 Speed:10000 Mtu:8901} {Name:b1741e5a58e70e6 MacAddress:6e:f2:3a:6e:a3:e3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:eb:1e:51 Speed:0 Mtu:9001} {Name:br-int MacAddress:3a:70:6c:79:0c:5a Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:eb:1e:51 Speed:0 Mtu:9001} {Name:f23c0dd1b15c4d1 MacAddress:8e:09:0c:43:a6:13 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:9a:a9:6a:4f:15:94 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:8c:f9:d2:ee:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164345344 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.245601    4497 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.245645    4497 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.245938    4497 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.245961    4497 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246102    4497 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246109    4497 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246119    4497 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246139    4497 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246243    4497 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246412    4497 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246515    4497 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246551    4497 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246561    4497 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246571    4497 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246581    4497 kubelet.go:405] "Adding apiserver pod source"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.246589    4497 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.248279    4497 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 16:22:50.249477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.248314    4497 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 16:22:50.250181 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.249606    4497 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 16:22:50.251667 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.251653    4497 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 16:22:50.251667 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.251666    4497 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 16:22:50.251767 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.251758    4497 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 16:22:50.251956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.251946    4497 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 16:22:50.251985 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.251975    4497 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 16:22:50.252070 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252061    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 16:22:50.252108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252076    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 16:22:50.252108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252082    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 16:22:50.252108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252088    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 16:22:50.252108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252094    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 16:22:50.252108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252099    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 16:22:50.252108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252106    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 16:22:50.252350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252112    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 16:22:50.252350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252119    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 16:22:50.252350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252125    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 16:22:50.252350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252134    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 16:22:50.252350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252143    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 16:22:50.252350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252166    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 16:22:50.252350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.252172    4497 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 16:22:50.255528 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.255512    4497 server.go:1263] "Started kubelet"
Oct 30 16:22:50.255662 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.255604    4497 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 16:22:50.256048 ip-10-0-136-74 systemd[1]: Started Kubernetes Kubelet.
Oct 30 16:22:50.256428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.256029    4497 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 16:22:50.256428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.256114    4497 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 16:22:50.256428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.256416    4497 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 16:22:50.257248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.257232    4497 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 16:22:50.259501 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.259480    4497 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 16:22:50.259587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.259482    4497 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 16:22:50.259637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.259598    4497 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:12 +0000 UTC" deadline="2028-05-29 03:38:51.690193033 +0000 UTC"
Oct 30 16:22:50.259684 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.259636    4497 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13835h16m1.430559708s"
Oct 30 16:22:50.259903 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:50.259880    4497 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-74.ec2.internal\" not found"
Oct 30 16:22:50.260060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.260046    4497 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 16:22:50.260060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.260058    4497 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 16:22:50.260164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.260143    4497 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.261280    4497 factory.go:55] Registering systemd factory
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.261301    4497 factory.go:223] Registration of the systemd container factory successfully
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262113    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" volumeName="kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-binary-copy" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262166    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8aea925-85ca-48f2-a6a3-03eefedf6022" volumeName="kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262201    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfe1d12e-6cb7-487f-a352-590c8fcf7775" volumeName="kubernetes.io/empty-dir/cfe1d12e-6cb7-487f-a352-590c8fcf7775-tmp-dir" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262220    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31bd08bb-58ea-4508-8e99-f4a1d9b552ad" volumeName="kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-tuned" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262231    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c365930-eba1-4476-8a6c-7efe851313a8" volumeName="kubernetes.io/configmap/3c365930-eba1-4476-8a6c-7efe851313a8-konnectivity-ca" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262251    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="529ebf7f-7783-4575-a1fd-5c672e410f6a" volumeName="kubernetes.io/configmap/529ebf7f-7783-4575-a1fd-5c672e410f6a-iptables-alerter-script" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262264    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" volumeName="kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262271    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="918257e5-108f-4407-a0a5-1b540606ff05" volumeName="kubernetes.io/projected/918257e5-108f-4407-a0a5-1b540606ff05-kube-api-access-bmcfr" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262291    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31bd08bb-58ea-4508-8e99-f4a1d9b552ad" volumeName="kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-tmp" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262305    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57050003-649e-4fad-ba51-9f1a534a859c" volumeName="kubernetes.io/configmap/57050003-649e-4fad-ba51-9f1a534a859c-serviceca" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262313    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d1b009-1219-4dba-97a5-32d9f88b1447" volumeName="kubernetes.io/projected/b6d1b009-1219-4dba-97a5-32d9f88b1447-kube-api-access-ppj6l" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262327    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8853655-ace7-46f7-89f4-8be79930cffa" volumeName="kubernetes.io/projected/b8853655-ace7-46f7-89f4-8be79930cffa-kube-api-access-n2tjs" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262336    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8853655-ace7-46f7-89f4-8be79930cffa" volumeName="kubernetes.io/secret/b8853655-ace7-46f7-89f4-8be79930cffa-cert" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262344    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a046e81-b1ab-4da2-ac50-7d06f2c30668" volumeName="kubernetes.io/empty-dir/3a046e81-b1ab-4da2-ac50-7d06f2c30668-tmp-dir" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262359    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a046e81-b1ab-4da2-ac50-7d06f2c30668" volumeName="kubernetes.io/projected/3a046e81-b1ab-4da2-ac50-7d06f2c30668-kube-api-access-2xmb7" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262392    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d1b009-1219-4dba-97a5-32d9f88b1447" volumeName="kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-daemon-config" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262403    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b88993de-a05b-40ff-a753-8b9949f5f6c4" volumeName="kubernetes.io/projected/b88993de-a05b-40ff-a753-8b9949f5f6c4-kube-api-access-s2d2w" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262429    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8aea925-85ca-48f2-a6a3-03eefedf6022" volumeName="kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262439    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="529ebf7f-7783-4575-a1fd-5c672e410f6a" volumeName="kubernetes.io/projected/529ebf7f-7783-4575-a1fd-5c672e410f6a-kube-api-access-9kctn" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262451    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83bc0294-07c9-4087-88cf-1cf891d84118" volumeName="kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-env-overrides" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262465    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="918257e5-108f-4407-a0a5-1b540606ff05" volumeName="kubernetes.io/empty-dir/918257e5-108f-4407-a0a5-1b540606ff05-data-volume" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262472    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bea838c0-50c6-4c52-a0ff-e1e019022677" volumeName="kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262478    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8aea925-85ca-48f2-a6a3-03eefedf6022" volumeName="kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262491    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfe1d12e-6cb7-487f-a352-590c8fcf7775" volumeName="kubernetes.io/projected/cfe1d12e-6cb7-487f-a352-590c8fcf7775-kube-api-access-m46dw" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262498    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31bd08bb-58ea-4508-8e99-f4a1d9b552ad" volumeName="kubernetes.io/projected/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-kube-api-access-vkglq" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262521    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83bc0294-07c9-4087-88cf-1cf891d84118" volumeName="kubernetes.io/projected/83bc0294-07c9-4087-88cf-1cf891d84118-kube-api-access-wc4gx" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262534    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" volumeName="kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262546    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8aea925-85ca-48f2-a6a3-03eefedf6022" volumeName="kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262570    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c365930-eba1-4476-8a6c-7efe851313a8" volumeName="kubernetes.io/secret/3c365930-eba1-4476-8a6c-7efe851313a8-agent-certs" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262583    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="536629a7-e02f-4d49-a9de-cccd0e39132e" volumeName="kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262591    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57050003-649e-4fad-ba51-9f1a534a859c" volumeName="kubernetes.io/projected/57050003-649e-4fad-ba51-9f1a534a859c-kube-api-access-9p46b" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262597    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83bc0294-07c9-4087-88cf-1cf891d84118" volumeName="kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-config" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262610    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6d1b009-1219-4dba-97a5-32d9f88b1447" volumeName="kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-cni-binary-copy" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262641    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfe1d12e-6cb7-487f-a352-590c8fcf7775" volumeName="kubernetes.io/configmap/cfe1d12e-6cb7-487f-a352-590c8fcf7775-config-volume" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262648    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee56683d-41c9-4d01-83c4-57c1c1e184d4" volumeName="kubernetes.io/projected/ee56683d-41c9-4d01-83c4-57c1c1e184d4-kube-api-access-8bmn6" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262662    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7eb87a-b4ac-4d73-8892-fb64193dbbd8" volumeName="kubernetes.io/projected/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kube-api-access-qmfr2" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262669    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83bc0294-07c9-4087-88cf-1cf891d84118" volumeName="kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262676    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83bc0294-07c9-4087-88cf-1cf891d84118" volumeName="kubernetes.io/secret/83bc0294-07c9-4087-88cf-1cf891d84118-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262687    4497 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" volumeName="kubernetes.io/projected/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-kube-api-access-jdvbj" seLinuxMountContext=""
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262699    4497 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262705    4497 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262918    4497 factory.go:153] Registering CRI-O factory
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262941    4497 factory.go:223] Registration of the crio container factory successfully
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.262996    4497 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.263046    4497 factory.go:103] Registering Raw factory
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.263057    4497 manager.go:1199] Started watching for new ooms in manager
Oct 30 16:22:50.265213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.263666    4497 manager.go:319] Starting recovery of all containers
Oct 30 16:22:50.290485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.290438    4497 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 16:22:50.291806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.291782    4497 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 16:22:50.291806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.291805    4497 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 16:22:50.291964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.291821    4497 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 16:22:50.291964 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:50.291854    4497 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 16:22:50.300203 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.300158    4497 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="7cbb50eaac6f92c79b2854d2ef9e079b960027b902def012926da6f410b3b279" exitCode=0
Oct 30 16:22:50.300334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.300218    4497 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="8b8d7de3cbf5daab09f07d87d29fc078031ca9c1b737e058b7f96d3a03137a13" exitCode=0
Oct 30 16:22:50.300334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.300228    4497 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="5f8c7f6d053ed4907559fc1422ec4ca11d334a669f9ae88899776588a1b371b3" exitCode=0
Oct 30 16:22:50.300334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.300237    4497 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="a5a4001ea523983a4277f5b5d2dc1e5f5fc12fa61e15ac834de1da810594e72d" exitCode=0
Oct 30 16:22:50.300334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.300245    4497 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="4d08ef3e4528d6fa9964545a1656bb0749d4d1cd7fe8ed14d19a73dd76f5a88c" exitCode=0
Oct 30 16:22:50.300334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.300252    4497 generic.go:356] "Generic (PLEG): container finished" podID="8b89a9fa-bf4d-44f6-9331-22a216a06f6d" containerID="e4daa5fda1466410e1ed370d6888de951bd2f8fe3cfd932afd0995b966dfa412" exitCode=0
Oct 30 16:22:50.306769 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.306746    4497 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 16:22:50.307072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.307045    4497 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:50.307204 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.307049    4497 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:50.307278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.307209    4497 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:50.312305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.312277    4497 generic.go:356] "Generic (PLEG): container finished" podID="a6861b6533e84603bf25f2e81fd4e489" containerID="d9057d79573e6d02c4e65dae9a6ef161f4c9c6affe419c7d837f7d028f62bb4b" exitCode=0
Oct 30 16:22:50.336532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.336457    4497 manager.go:324] Recovery completed
Oct 30 16:22:50.345520 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.345492    4497 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 16:22:50.352783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352758    4497 cpu_manager.go:226] "Starting" policy="none"
Oct 30 16:22:50.352783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352776    4497 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 16:22:50.352966 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352796    4497 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 16:22:50.352966 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352940    4497 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 16:22:50.352966 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352953    4497 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 16:22:50.353074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352969    4497 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 16:22:50.353074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352975    4497 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 16:22:50.353074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352981    4497 policy_none.go:50] "Start"
Oct 30 16:22:50.353074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.352993    4497 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 16:22:50.353074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.353005    4497 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 16:22:50.353228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.353137    4497 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 16:22:50.353228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.353144    4497 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 16:22:50.353228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.353150    4497 policy_none.go:44] "Start"
Oct 30 16:22:50.357060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.357028    4497 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 16:22:50.359826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.359813    4497 manager.go:341] "Starting Device Plugin manager"
Oct 30 16:22:50.360357 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:50.359842    4497 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 16:22:50.360357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.360357    4497 server.go:87] "Starting device plugin registration server"
Oct 30 16:22:50.360639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.360626    4497 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 16:22:50.360727 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.360638    4497 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 16:22:50.360765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.360740    4497 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 16:22:50.360884 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.360846    4497 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 16:22:50.360884 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.360893    4497 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 16:22:50.360984 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.360900    4497 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 16:22:50.361078 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.360953    4497 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:22:50.360882407Z","UUID":"f660019d-1541-49f1-9f82-93df62035b39","Handler":null,"Name":"","Endpoint":""}
Oct 30 16:22:50.362237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.362220    4497 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 16:22:50.362237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.362242    4497 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 16:22:50.362375 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:50.362310    4497 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 16:22:50.392496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.392444    4497 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"]
Oct 30 16:22:50.393070 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.393019    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal" event={"ID":"a6861b6533e84603bf25f2e81fd4e489","Type":"ContainerStarted","Data":"cc595e8d0c269d2f1a536e41dc3ae8797dfc1d5c3d9103f4755f75abd15927c9"}
Oct 30 16:22:50.393215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.393078    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal" event={"ID":"a6861b6533e84603bf25f2e81fd4e489","Type":"ContainerDied","Data":"d9057d79573e6d02c4e65dae9a6ef161f4c9c6affe419c7d837f7d028f62bb4b"}
Oct 30 16:22:50.393215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.393101    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal" event={"ID":"a6861b6533e84603bf25f2e81fd4e489","Type":"ContainerStarted","Data":"7ab6d0bd18409d2c903badb6074cca41ba8bcf67965b2f5cdca4078b5ec9ad34"}
Oct 30 16:22:50.393215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.393135    4497 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.393318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.393245    4497 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.393318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.393134    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal" event={"ID":"041057424ec306b79ef7363e62fcbac0","Type":"ContainerStarted","Data":"8795e8d324639119a918af97447a511832d64bde9c9de9380e0eb5036f22073a"}
Oct 30 16:22:50.393390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.393354    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal" event={"ID":"041057424ec306b79ef7363e62fcbac0","Type":"ContainerStarted","Data":"ec42e889039300d4ce8157ac0177cafa75e1397bb94c07394e66beaedbc228dd"}
Oct 30 16:22:50.415419 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.415390    4497 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:22:50.415584 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:50.415454    4497 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-74.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.415734 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.415719    4497 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:22:50.415774 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:50.415757    4497 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.460938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.460903    4497 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 16:22:50.462389 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.462362    4497 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 16:22:50.462513 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.462404    4497 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 16:22:50.462513 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.462420    4497 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-74.ec2.internal" event="NodeHasSufficientPID"
Oct 30 16:22:50.462513 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.462466    4497 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.492391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.492354    4497 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.492548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.492433    4497 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.508007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.507967    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041057424ec306b79ef7363e62fcbac0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-74.ec2.internal\" (UID: \"041057424ec306b79ef7363e62fcbac0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.508007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.508002    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.508227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.508022    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.608995 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.608894    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.608995 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.608940    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.608995 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.608964    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041057424ec306b79ef7363e62fcbac0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-74.ec2.internal\" (UID: \"041057424ec306b79ef7363e62fcbac0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.609251 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.609019    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/041057424ec306b79ef7363e62fcbac0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-74.ec2.internal\" (UID: \"041057424ec306b79ef7363e62fcbac0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.609251 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.609022    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:50.609251 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:50.609020    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a6861b6533e84603bf25f2e81fd4e489-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" (UID: \"a6861b6533e84603bf25f2e81fd4e489\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:51.247344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.247309    4497 apiserver.go:52] "Watching apiserver"
Oct 30 16:22:51.289706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.289673    4497 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 16:22:51.290135 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.290114    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rc86k","openshift-insights/insights-runtime-extractor-6scbv","openshift-multus/network-metrics-daemon-pv5p7","openshift-cluster-node-tuning-operator/tuned-pz9z8","openshift-dns/node-resolver-2lxwn","openshift-network-operator/iptables-alerter-jwpjd","openshift-insights/periodic-gathering-jj2nm-v6brc","openshift-multus/multus-additional-cni-plugins-jv959","kube-system/global-pull-secret-syncer-x4pp5","kube-system/konnectivity-agent-j6fh2","openshift-ovn-kubernetes/ovnkube-node-66645","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg","openshift-network-diagnostics/network-check-target-htgvf","kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal","openshift-multus/multus-k545t","openshift-ingress-canary/ingress-canary-fszqg","openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76","openshift-dns/dns-default-ts8b4"]
Oct 30 16:22:51.291303 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.291278    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:51.294952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.294930    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.295514 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.295496    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.303962 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.303938    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 16:22:51.304112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.303944    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.304842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.304812    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 16:22:51.304934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.304839    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.305110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.305083    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.305267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.305244    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.305468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.305445    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 16:22:51.312039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.312013    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.312208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.312067    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.312208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.312206    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 16:22:51.312325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.312220    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 16:22:51.312401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.312385    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 16:22:51.312444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.312430    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 16:22:51.312484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.312475    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.317681 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.317658    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.321404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.321386    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 16:22:51.321565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.321547    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 16:22:51.322623 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.322605    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 16:22:51.322918 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.322902    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kf5z\""
Oct 30 16:22:51.323004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.322905    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vhqb4\""
Oct 30 16:22:51.323057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323015    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 16:22:51.323057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323026    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hpqxb\""
Oct 30 16:22:51.323057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323038    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lpvkt\""
Oct 30 16:22:51.323173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323083    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m9x9f\""
Oct 30 16:22:51.323173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323166    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lf54z\""
Oct 30 16:22:51.323279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323177    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 16:22:51.323279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323234    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h4vt2\""
Oct 30 16:22:51.323279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323238    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tdd2t\""
Oct 30 16:22:51.323279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323255    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9txpl\""
Oct 30 16:22:51.323420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323291    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6852\""
Oct 30 16:22:51.323420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323308    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-86hj4\""
Oct 30 16:22:51.323420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.323316    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5n7vr\""
Oct 30 16:22:51.324014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.324000    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bls9m\""
Oct 30 16:22:51.334741 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.334721    4497 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:51.334872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.334859    4497 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:51.353469 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.353438    4497 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:22:51.353581 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:51.353492    4497 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-74.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:51.353992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.353978    4497 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 16:22:51.354031 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:51.354014    4497 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-74.ec2.internal"
Oct 30 16:22:51.362159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.362128    4497 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 16:22:51.380369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380333    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 16:22:51.380369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380367    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.380617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380377    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.380617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380424    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.380754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380653    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 16:22:51.380754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380674    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.380754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380685    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.380754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380705    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.381013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380920    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 16:22:51.381013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380943    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.381013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380956    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 16:22:51.381013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380959    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 16:22:51.381013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380978    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.381013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380960    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 16:22:51.381013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.380993    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.381013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.381005    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.381398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.381022    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.381398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.381046    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:22:51.381398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.381070    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.381398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.381307    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 16:22:51.381803 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.381789    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 16:22:51.412167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.411806    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-multus-certs\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.412167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.411838    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfr2\" (UniqueName: \"kubernetes.io/projected/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kube-api-access-qmfr2\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.412167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.411859    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcfr\" (UniqueName: \"kubernetes.io/projected/918257e5-108f-4407-a0a5-1b540606ff05-kube-api-access-bmcfr\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.412167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.411877    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-env-overrides\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.412167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.411949    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.412167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.411988    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-systemd\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412198    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-host\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412272    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-env-overrides\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412279    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmn6\" (UniqueName: \"kubernetes.io/projected/ee56683d-41c9-4d01-83c4-57c1c1e184d4-kube-api-access-8bmn6\") pod \"migrator-66df5d496b-pmd76\" (UID: \"ee56683d-41c9-4d01-83c4-57c1c1e184d4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412333    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529ebf7f-7783-4575-a1fd-5c672e410f6a-host-slash\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412354    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412413    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a046e81-b1ab-4da2-ac50-7d06f2c30668-hosts-file\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412438    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmb7\" (UniqueName: \"kubernetes.io/projected/3a046e81-b1ab-4da2-ac50-7d06f2c30668-kube-api-access-2xmb7\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412454    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-multus\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412480    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412516    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c365930-eba1-4476-8a6c-7efe851313a8-agent-certs\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:51.412583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412576    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57050003-649e-4fad-ba51-9f1a534a859c-host\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412667    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412697    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/918257e5-108f-4407-a0a5-1b540606ff05-crio-socket\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412725    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-slash\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412757    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-netns\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412792    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-modprobe-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412823    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412886    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a046e81-b1ab-4da2-ac50-7d06f2c30668-tmp-dir\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412913    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3c365930-eba1-4476-8a6c-7efe851313a8-agent-certs\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412947    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-socket-dir-parent\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412967    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-hostroot\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.412989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.412990    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a046e81-b1ab-4da2-ac50-7d06f2c30668-tmp-dir\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413007    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413035    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-systemd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413063    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/529ebf7f-7783-4575-a1fd-5c672e410f6a-iptables-alerter-script\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413084    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413104    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-k8s-cni-cncf-io\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413182    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-bin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413215    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413235    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2d2w\" (UniqueName: \"kubernetes.io/projected/b88993de-a05b-40ff-a753-8b9949f5f6c4-kube-api-access-s2d2w\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413259    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413303    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/529ebf7f-7783-4575-a1fd-5c672e410f6a-iptables-alerter-script\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413309    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvbj\" (UniqueName: \"kubernetes.io/projected/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-kube-api-access-jdvbj\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413348    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-system-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413374    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-cnibin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413425    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-kubelet\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.413443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413436    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413455    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413479    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-kubelet\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413506    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413532    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413556    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-conf-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413605    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-etc-kubernetes\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413638    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413667    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413706    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57050003-649e-4fad-ba51-9f1a534a859c-serviceca\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413732    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/536629a7-e02f-4d49-a9de-cccd0e39132e-original-pull-secret\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413795    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413796    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-device-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413829    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptftn\" (UniqueName: \"kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413862    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413866    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-var-lib-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413873    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413901    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-netd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.413905    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/57050003-649e-4fad-ba51-9f1a534a859c-serviceca\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414003    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cnibin\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.414038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414044    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8853655-ace7-46f7-89f4-8be79930cffa-cert\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414092    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414170    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414215    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-systemd-units\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414238    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-config\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414247    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8853655-ace7-46f7-89f4-8be79930cffa-cert\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414255    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414291    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-sys\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414309    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-tmp\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414326    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414342    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4gx\" (UniqueName: \"kubernetes.io/projected/83bc0294-07c9-4087-88cf-1cf891d84118-kube-api-access-wc4gx\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414472    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-run\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414485    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-config\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414572    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-tmp\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414577    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-var-lib-kubelet\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414606    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe1d12e-6cb7-487f-a352-590c8fcf7775-config-volume\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414640    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe1d12e-6cb7-487f-a352-590c8fcf7775-tmp-dir\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.414702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414666    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46dw\" (UniqueName: \"kubernetes.io/projected/cfe1d12e-6cb7-487f-a352-590c8fcf7775-kube-api-access-m46dw\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414784    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-netns\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414790    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe1d12e-6cb7-487f-a352-590c8fcf7775-config-volume\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414803    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfe1d12e-6cb7-487f-a352-590c8fcf7775-tmp-dir\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414881    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414915    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/918257e5-108f-4407-a0a5-1b540606ff05-data-volume\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414956    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.414990    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415020    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/918257e5-108f-4407-a0a5-1b540606ff05-data-volume\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415105    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415109    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-conf\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415145    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-tuned\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415203    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83bc0294-07c9-4087-88cf-1cf891d84118-ovn-node-metrics-cert\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415219    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-tuned\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415229    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-cni-binary-copy\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.415325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415258    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-daemon-config\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415354    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83bc0294-07c9-4087-88cf-1cf891d84118-ovn-node-metrics-cert\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415374    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415393    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-node-log\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415407    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-kubernetes\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415421    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-kubelet-config\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415454    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-daemon-config\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415459    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p46b\" (UniqueName: \"kubernetes.io/projected/57050003-649e-4fad-ba51-9f1a534a859c-kube-api-access-9p46b\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415486    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kctn\" (UniqueName: \"kubernetes.io/projected/529ebf7f-7783-4575-a1fd-5c672e410f6a-kube-api-access-9kctn\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415493    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6d1b009-1219-4dba-97a5-32d9f88b1447-cni-binary-copy\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415513    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppj6l\" (UniqueName: \"kubernetes.io/projected/b6d1b009-1219-4dba-97a5-32d9f88b1447-kube-api-access-ppj6l\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415533    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415708    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-etc-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415728    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-log-socket\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415750    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c365930-eba1-4476-8a6c-7efe851313a8-konnectivity-ca\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415827    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysconfig\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415861    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-dbus\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:51.415935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415906    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3c365930-eba1-4476-8a6c-7efe851313a8-konnectivity-ca\") pod \"konnectivity-agent-j6fh2\" (UID: \"3c365930-eba1-4476-8a6c-7efe851313a8\") " pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415949    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.415981    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.416005    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-ovn\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.416033    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-bin\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.416051    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-script-lib\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.416065    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-lib-modules\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.416151    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkglq\" (UniqueName: \"kubernetes.io/projected/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-kube-api-access-vkglq\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.416178    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tjs\" (UniqueName: \"kubernetes.io/projected/b8853655-ace7-46f7-89f4-8be79930cffa-kube-api-access-n2tjs\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:51.416584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.416288    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83bc0294-07c9-4087-88cf-1cf891d84118-ovnkube-script-lib\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.419797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.419743    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 16:22:51.419797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.419751    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 16:22:51.419797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.419772    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjhbz\""
Oct 30 16:22:51.419797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.419773    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 16:22:51.420026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.419814    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z9zpr\""
Oct 30 16:22:51.420026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.419928    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-65hn8\""
Oct 30 16:22:51.420026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.419972    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6n6gp\""
Oct 30 16:22:51.431754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.431730    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmn6\" (UniqueName: \"kubernetes.io/projected/ee56683d-41c9-4d01-83c4-57c1c1e184d4-kube-api-access-8bmn6\") pod \"migrator-66df5d496b-pmd76\" (UID: \"ee56683d-41c9-4d01-83c4-57c1c1e184d4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-pmd76"
Oct 30 16:22:51.432025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.432006    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfr2\" (UniqueName: \"kubernetes.io/projected/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kube-api-access-qmfr2\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.432299 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.432284    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcfr\" (UniqueName: \"kubernetes.io/projected/918257e5-108f-4407-a0a5-1b540606ff05-kube-api-access-bmcfr\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.432724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.432707    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2d2w\" (UniqueName: \"kubernetes.io/projected/b88993de-a05b-40ff-a753-8b9949f5f6c4-kube-api-access-s2d2w\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:51.432979 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.432949    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvbj\" (UniqueName: \"kubernetes.io/projected/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-kube-api-access-jdvbj\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.433268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.433253    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmb7\" (UniqueName: \"kubernetes.io/projected/3a046e81-b1ab-4da2-ac50-7d06f2c30668-kube-api-access-2xmb7\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:51.433653 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.433638    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptftn\" (UniqueName: \"kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn\") pod \"periodic-gathering-jj2nm-v6brc\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") " pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:22:51.434433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.434412    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4gx\" (UniqueName: \"kubernetes.io/projected/83bc0294-07c9-4087-88cf-1cf891d84118-kube-api-access-wc4gx\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.435238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.435222    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgv9\" (UniqueName: \"kubernetes.io/projected/bea838c0-50c6-4c52-a0ff-e1e019022677-kube-api-access-4qgv9\") pod \"network-check-target-htgvf\" (UID: \"bea838c0-50c6-4c52-a0ff-e1e019022677\") " pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:51.436845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.436828    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46dw\" (UniqueName: \"kubernetes.io/projected/cfe1d12e-6cb7-487f-a352-590c8fcf7775-kube-api-access-m46dw\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.437462 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.437438    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkglq\" (UniqueName: \"kubernetes.io/projected/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-kube-api-access-vkglq\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.437650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.437603    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p46b\" (UniqueName: \"kubernetes.io/projected/57050003-649e-4fad-ba51-9f1a534a859c-kube-api-access-9p46b\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:51.437789 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.437774    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kctn\" (UniqueName: \"kubernetes.io/projected/529ebf7f-7783-4575-a1fd-5c672e410f6a-kube-api-access-9kctn\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:51.437823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.437773    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tjs\" (UniqueName: \"kubernetes.io/projected/b8853655-ace7-46f7-89f4-8be79930cffa-kube-api-access-n2tjs\") pod \"ingress-canary-fszqg\" (UID: \"b8853655-ace7-46f7-89f4-8be79930cffa\") " pod="openshift-ingress-canary/ingress-canary-fszqg"
Oct 30 16:22:51.437899 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.437886    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppj6l\" (UniqueName: \"kubernetes.io/projected/b6d1b009-1219-4dba-97a5-32d9f88b1447-kube-api-access-ppj6l\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.516557 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516523    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-host\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.516557 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516556    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529ebf7f-7783-4575-a1fd-5c672e410f6a-host-slash\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516572    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516587    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a046e81-b1ab-4da2-ac50-7d06f2c30668-hosts-file\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516603    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-multus\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516618    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516641    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57050003-649e-4fad-ba51-9f1a534a859c-host\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516664    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516666    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529ebf7f-7783-4575-a1fd-5c672e410f6a-host-slash\") pod \"iptables-alerter-jwpjd\" (UID: \"529ebf7f-7783-4575-a1fd-5c672e410f6a\") " pod="openshift-network-operator/iptables-alerter-jwpjd"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516685    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-multus\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516705    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57050003-649e-4fad-ba51-9f1a534a859c-host\") pod \"node-ca-rc86k\" (UID: \"57050003-649e-4fad-ba51-9f1a534a859c\") " pod="openshift-image-registry/node-ca-rc86k"
Oct 30 16:22:51.516783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516711    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a046e81-b1ab-4da2-ac50-7d06f2c30668-hosts-file\") pod \"node-resolver-2lxwn\" (UID: \"3a046e81-b1ab-4da2-ac50-7d06f2c30668\") " pod="openshift-dns/node-resolver-2lxwn"
Oct 30 16:22:51.517244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516691    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/918257e5-108f-4407-a0a5-1b540606ff05-crio-socket\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.517244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516664    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-host\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.517244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516729    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.517244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517050    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-slash\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.517244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517092    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-netns\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.517244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517133    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-modprobe-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.517244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517169    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/918257e5-108f-4407-a0a5-1b540606ff05-crio-socket\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.517244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.516663    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517278    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-slash\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517363    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-modprobe-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517338    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-run-netns\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517392    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-socket-dir-parent\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517436    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-hostroot\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517445    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-socket-dir-parent\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517474    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517498    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-hostroot\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517506    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-systemd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517554    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.517590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517561    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-systemd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517593    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-k8s-cni-cncf-io\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517627    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/918257e5-108f-4407-a0a5-1b540606ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517631    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-bin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517676    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-cni-bin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517705    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-k8s-cni-cncf-io\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517754    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517793    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-system-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517817    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-cnibin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517869    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-registration-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517923    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-system-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517958    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-kubelet\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.517994    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-kubelet\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518027    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.518054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518056    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-conf-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518086    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-etc-kubernetes\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518120    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-device-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518154    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-var-lib-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518197    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-netd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518225    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cnibin\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518257    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518286    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518314    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-systemd-units\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518344    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518369    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-sys\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518402    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-run\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518430    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-var-lib-kubelet\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518463    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-netns\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518491    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518517    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518545    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-conf\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518578    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518608    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-node-log\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.518632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518632    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-kubernetes\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518661    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-kubelet-config\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518696    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518725    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-etc-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518754    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-log-socket\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518782    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysconfig\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518812    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-dbus\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518843    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518874    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518901    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-ovn\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518932    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-bin\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.518967    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-lib-modules\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519002    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-multus-certs\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519036    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519065    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-systemd\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519180    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-systemd\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519258    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-cnibin\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519308    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-var-lib-kubelet\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519356    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-kubelet\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.519416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519405    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-system-cni-dir\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519455    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-conf-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519497    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-etc-kubernetes\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519551    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-device-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519596    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-var-lib-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519638    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-netd\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519681    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b89a9fa-bf4d-44f6-9331-22a216a06f6d-cnibin\") pod \"multus-additional-cni-plugins-jv959\" (UID: \"8b89a9fa-bf4d-44f6-9331-22a216a06f6d\") " pod="openshift-multus/multus-additional-cni-plugins-jv959"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519734    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-socket-dir\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519792    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-sys-fs\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.519833    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-systemd-units\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:51.519943    4497 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:51.520058    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:52.02003556 +0000 UTC m=+1.873913817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : secret "metrics-daemon-secret" not found
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:51.520073    4497 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520114    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-etc-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520164    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-log-socket\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.520211 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:51.520209    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:52.020149586 +0000 UTC m=+1.874027847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520241    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysconfig\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520267    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-sys\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520305    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-dbus\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520322    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-run\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520392    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-ovn\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520449    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-cni-bin\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520482    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-conf\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520539    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-var-lib-kubelet\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520559    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-lib-modules\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520593    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1a7eb87a-b4ac-4d73-8892-fb64193dbbd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-mg7tg\" (UID: \"1a7eb87a-b4ac-4d73-8892-fb64193dbbd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mg7tg"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520638    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-multus-certs\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520703    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-run-openvswitch\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520704    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-sysctl-d\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.520850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520734    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/536629a7-e02f-4d49-a9de-cccd0e39132e-kubelet-config\") pod \"global-pull-secret-syncer-x4pp5\" (UID: \"536629a7-e02f-4d49-a9de-cccd0e39132e\") " pod="kube-system/global-pull-secret-syncer-x4pp5"
Oct 30 16:22:51.521440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520859    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.521440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520936    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-multus-cni-dir\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.521440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.520981    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6d1b009-1219-4dba-97a5-32d9f88b1447-host-run-netns\") pod \"multus-k545t\" (UID: \"b6d1b009-1219-4dba-97a5-32d9f88b1447\") " pod="openshift-multus/multus-k545t"
Oct 30 16:22:51.521440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.521028    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83bc0294-07c9-4087-88cf-1cf891d84118-node-log\") pod \"ovnkube-node-66645\" (UID: \"83bc0294-07c9-4087-88cf-1cf891d84118\") " pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:51.521440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.521065    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31bd08bb-58ea-4508-8e99-f4a1d9b552ad-etc-kubernetes\") pod \"tuned-pz9z8\" (UID: \"31bd08bb-58ea-4508-8e99-f4a1d9b552ad\") " pod="openshift-cluster-node-tuning-operator/tuned-pz9z8"
Oct 30 16:22:51.521440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.521086    4497 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 16:22:51.523325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.523307    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cfe1d12e-6cb7-487f-a352-590c8fcf7775-metrics-tls\") pod \"dns-default-ts8b4\" (UID: \"cfe1d12e-6cb7-487f-a352-590c8fcf7775\") " pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.595201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.595146    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:51.925070 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.924988    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ts8b4"]
Oct 30 16:22:51.931610 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:51.931556    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe1d12e_6cb7_487f_a352_590c8fcf7775.slice/crio-ce333f18b8620565a5cb85c3fd6a83201c6aadf4eb1b5574c03768e1a2c6e13d WatchSource:0}: Error finding container ce333f18b8620565a5cb85c3fd6a83201c6aadf4eb1b5574c03768e1a2c6e13d: Status 404 returned error can't find the container with id ce333f18b8620565a5cb85c3fd6a83201c6aadf4eb1b5574c03768e1a2c6e13d
Oct 30 16:22:51.933648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:51.933632    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:22:52.023139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:52.023104    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:52.023300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:52.023150    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:52.023300 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:52.023274    4497 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:52.023380 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:52.023323    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:53.023311552 +0000 UTC m=+2.877189808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:52.023380 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:52.023274    4497 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 16:22:52.023442 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:52.023403    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs podName:b88993de-a05b-40ff-a753-8b9949f5f6c4 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:53.02338882 +0000 UTC m=+2.877267076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs") pod "network-metrics-daemon-pv5p7" (UID: "b88993de-a05b-40ff-a753-8b9949f5f6c4") : secret "metrics-daemon-secret" not found
Oct 30 16:22:52.047647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:52.047621    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:52.338656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:52.338617    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ts8b4" event={"ID":"cfe1d12e-6cb7-487f-a352-590c8fcf7775","Type":"ContainerStarted","Data":"ce333f18b8620565a5cb85c3fd6a83201c6aadf4eb1b5574c03768e1a2c6e13d"}
Oct 30 16:22:53.032532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:53.032494    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:53.032746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:53.032557    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:53.032746 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:53.032653    4497 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:53.032746 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:53.032719    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:55.032704539 +0000 UTC m=+4.886582782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:53.034818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:53.034793    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88993de-a05b-40ff-a753-8b9949f5f6c4-metrics-certs\") pod \"network-metrics-daemon-pv5p7\" (UID: \"b88993de-a05b-40ff-a753-8b9949f5f6c4\") " pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:53.092201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:53.092155    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pv5p7"
Oct 30 16:22:53.401213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:53.401109    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pv5p7"]
Oct 30 16:22:53.408009 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:53.407968    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88993de_a05b_40ff_a753_8b9949f5f6c4.slice/crio-7563ed789739a4aaac0c7a151282e46ca55afec33b5dc3f64337d0eab6ce2c11 WatchSource:0}: Error finding container 7563ed789739a4aaac0c7a151282e46ca55afec33b5dc3f64337d0eab6ce2c11: Status 404 returned error can't find the container with id 7563ed789739a4aaac0c7a151282e46ca55afec33b5dc3f64337d0eab6ce2c11
Oct 30 16:22:54.346181 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:54.346126    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pv5p7" event={"ID":"b88993de-a05b-40ff-a753-8b9949f5f6c4","Type":"ContainerStarted","Data":"7563ed789739a4aaac0c7a151282e46ca55afec33b5dc3f64337d0eab6ce2c11"}
Oct 30 16:22:54.348331 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:54.348303    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ts8b4" event={"ID":"cfe1d12e-6cb7-487f-a352-590c8fcf7775","Type":"ContainerStarted","Data":"a806f79b8a32a8f7f5d2658b01e95d523097818522f65f7028ca35d4f93ec569"}
Oct 30 16:22:54.348497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:54.348357    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ts8b4" event={"ID":"cfe1d12e-6cb7-487f-a352-590c8fcf7775","Type":"ContainerStarted","Data":"86374cdd4c6b59c0f2563f6d16432bd7c4577b3b70d322c14d0f1125c633b7a2"}
Oct 30 16:22:54.348497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:54.348466    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:22:54.540583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:54.540546    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:22:54.540918 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:54.540760    4497 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 16:22:54.664295 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:54.664213    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ts8b4_cfe1d12e-6cb7-487f-a352-590c8fcf7775/dns/0.log"
Oct 30 16:22:54.864923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:54.864891    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ts8b4_cfe1d12e-6cb7-487f-a352-590c8fcf7775/kube-rbac-proxy/0.log"
Oct 30 16:22:55.050386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:55.050351    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:55.050556 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:55.050514    4497 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:55.050599 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:22:55.050582    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls podName:918257e5-108f-4407-a0a5-1b540606ff05 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:59.050562779 +0000 UTC m=+8.904441022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6scbv" (UID: "918257e5-108f-4407-a0a5-1b540606ff05") : secret "insights-runtime-extractor-tls" not found
Oct 30 16:22:55.064594 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:55.064566    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2lxwn_3a046e81-b1ab-4da2-ac50-7d06f2c30668/dns-node-resolver/0.log"
Oct 30 16:22:55.353011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:55.352928    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pv5p7" event={"ID":"b88993de-a05b-40ff-a753-8b9949f5f6c4","Type":"ContainerStarted","Data":"2efe5822fc27d16c9d515083c7137f7d588de2fc1692c626a970bb0eaa872556"}
Oct 30 16:22:55.353011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:55.352990    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pv5p7" event={"ID":"b88993de-a05b-40ff-a753-8b9949f5f6c4","Type":"ContainerStarted","Data":"44bba24c313673cd53196f303cd829613a2a501cd51ddd6ea4250b842a7335d7"}
Oct 30 16:22:56.467853 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:56.467824    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rc86k_57050003-649e-4fad-ba51-9f1a534a859c/node-ca/0.log"
Oct 30 16:22:56.864376 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:56.864348    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fszqg_b8853655-ace7-46f7-89f4-8be79930cffa/serve-healthcheck-canary/0.log"
Oct 30 16:22:56.888392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:56.888362    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:22:57.664649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:57.664621    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-pmd76_ee56683d-41c9-4d01-83c4-57c1c1e184d4/migrator/0.log"
Oct 30 16:22:59.077195 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:59.077158    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:59.079529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:59.079504    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/918257e5-108f-4407-a0a5-1b540606ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6scbv\" (UID: \"918257e5-108f-4407-a0a5-1b540606ff05\") " pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:59.096430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:59.096399    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6scbv"
Oct 30 16:22:59.406108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:22:59.406022    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6scbv"]
Oct 30 16:22:59.412298 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:22:59.412266    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918257e5_108f_4407_a0a5_1b540606ff05.slice/crio-47928ff205dc9e7bfad49f8c9530cca535bfca5fff7e7e274875b56b08919330 WatchSource:0}: Error finding container 47928ff205dc9e7bfad49f8c9530cca535bfca5fff7e7e274875b56b08919330: Status 404 returned error can't find the container with id 47928ff205dc9e7bfad49f8c9530cca535bfca5fff7e7e274875b56b08919330
Oct 30 16:23:00.371682 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:00.371645    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6scbv" event={"ID":"918257e5-108f-4407-a0a5-1b540606ff05","Type":"ContainerStarted","Data":"1417935329f98a8cb6b63880c1657cb5d9f5d1c02af74b012aa879b447ed06a5"}
Oct 30 16:23:00.372079 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:00.371719    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6scbv" event={"ID":"918257e5-108f-4407-a0a5-1b540606ff05","Type":"ContainerStarted","Data":"0a22d2b7aeb36e59940536625d92523a1f9bb936142f19979addbc401f438fe7"}
Oct 30 16:23:00.372079 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:00.371749    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6scbv" event={"ID":"918257e5-108f-4407-a0a5-1b540606ff05","Type":"ContainerStarted","Data":"47928ff205dc9e7bfad49f8c9530cca535bfca5fff7e7e274875b56b08919330"}
Oct 30 16:23:02.378925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:02.378889    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6scbv" event={"ID":"918257e5-108f-4407-a0a5-1b540606ff05","Type":"ContainerStarted","Data":"e5f32788c224b6550258333a7c78501c0cd5a6000a160b118f3cbabb9f3ef18c"}
Oct 30 16:23:04.355688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:04.355660    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ts8b4"
Oct 30 16:23:05.819351 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:05.819319    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7h5xd"]
Oct 30 16:23:05.821310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:05.821284    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:05.835270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:05.835240    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-n78rc\""
Oct 30 16:23:05.835417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:05.835402    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 16:23:05.845062 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:05.845034    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 16:23:05.858266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:05.858237    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7h5xd"]
Oct 30 16:23:05.925415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:05.925380    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21066a6d-880f-4d1d-ad2b-9c143cd219af-tmp\") pod \"downloads-6dd789c8b7-7h5xd\" (UID: \"21066a6d-880f-4d1d-ad2b-9c143cd219af\") " pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:05.925584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:05.925425    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5sxx\" (UniqueName: \"kubernetes.io/projected/21066a6d-880f-4d1d-ad2b-9c143cd219af-kube-api-access-g5sxx\") pod \"downloads-6dd789c8b7-7h5xd\" (UID: \"21066a6d-880f-4d1d-ad2b-9c143cd219af\") " pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:06.026139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:06.026097    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5sxx\" (UniqueName: \"kubernetes.io/projected/21066a6d-880f-4d1d-ad2b-9c143cd219af-kube-api-access-g5sxx\") pod \"downloads-6dd789c8b7-7h5xd\" (UID: \"21066a6d-880f-4d1d-ad2b-9c143cd219af\") " pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:06.026349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:06.026162    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21066a6d-880f-4d1d-ad2b-9c143cd219af-tmp\") pod \"downloads-6dd789c8b7-7h5xd\" (UID: \"21066a6d-880f-4d1d-ad2b-9c143cd219af\") " pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:06.026582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:06.026558    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21066a6d-880f-4d1d-ad2b-9c143cd219af-tmp\") pod \"downloads-6dd789c8b7-7h5xd\" (UID: \"21066a6d-880f-4d1d-ad2b-9c143cd219af\") " pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:06.049783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:06.049751    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5sxx\" (UniqueName: \"kubernetes.io/projected/21066a6d-880f-4d1d-ad2b-9c143cd219af-kube-api-access-g5sxx\") pod \"downloads-6dd789c8b7-7h5xd\" (UID: \"21066a6d-880f-4d1d-ad2b-9c143cd219af\") " pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:06.134008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:06.133923    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:06.457220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:06.457109    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7h5xd"]
Oct 30 16:23:06.465413 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:23:06.465384    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21066a6d_880f_4d1d_ad2b_9c143cd219af.slice/crio-60ff9b7951e9bc0592a5ac7bb2cc9af47aef53a1ebfbdd1821e20e40fa7f248f WatchSource:0}: Error finding container 60ff9b7951e9bc0592a5ac7bb2cc9af47aef53a1ebfbdd1821e20e40fa7f248f: Status 404 returned error can't find the container with id 60ff9b7951e9bc0592a5ac7bb2cc9af47aef53a1ebfbdd1821e20e40fa7f248f
Oct 30 16:23:07.396092 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:07.396051    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7h5xd" event={"ID":"21066a6d-880f-4d1d-ad2b-9c143cd219af","Type":"ContainerStarted","Data":"60ff9b7951e9bc0592a5ac7bb2cc9af47aef53a1ebfbdd1821e20e40fa7f248f"}
Oct 30 16:23:11.026273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:11.026237    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j6fh2"
Oct 30 16:23:12.802549 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.802514    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"]
Oct 30 16:23:12.805004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.804979    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:12.826979 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.826915    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c7glp\""
Oct 30 16:23:12.826979 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.826942    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 16:23:12.827167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.827050    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 16:23:12.827167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.827160    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 16:23:12.827276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.827161    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 16:23:12.827276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.827254    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 16:23:12.837692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.837654    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"]
Oct 30 16:23:12.972625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.972589    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ql4\" (UniqueName: \"kubernetes.io/projected/5f171456-2c42-41ef-b11a-352f6d543e37-kube-api-access-f9ql4\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:12.972820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.972636    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f171456-2c42-41ef-b11a-352f6d543e37-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:12.972820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.972669    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f171456-2c42-41ef-b11a-352f6d543e37-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:12.972820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:12.972814    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f171456-2c42-41ef-b11a-352f6d543e37-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.074152 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.074066    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f171456-2c42-41ef-b11a-352f6d543e37-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.074152 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.074134    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f171456-2c42-41ef-b11a-352f6d543e37-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.074404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.074170    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ql4\" (UniqueName: \"kubernetes.io/projected/5f171456-2c42-41ef-b11a-352f6d543e37-kube-api-access-f9ql4\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.074404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.074206    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f171456-2c42-41ef-b11a-352f6d543e37-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.074871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.074842    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f171456-2c42-41ef-b11a-352f6d543e37-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.076822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.076802    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5f171456-2c42-41ef-b11a-352f6d543e37-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.076942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.076923    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5f171456-2c42-41ef-b11a-352f6d543e37-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.095582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.095556    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ql4\" (UniqueName: \"kubernetes.io/projected/5f171456-2c42-41ef-b11a-352f6d543e37-kube-api-access-f9ql4\") pod \"prometheus-operator-9cd89b74c-8vg7x\" (UID: \"5f171456-2c42-41ef-b11a-352f6d543e37\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.119333 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.119299    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"
Oct 30 16:23:13.424922 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.424818    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x"]
Oct 30 16:23:13.430979 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:23:13.430945    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f171456_2c42_41ef_b11a_352f6d543e37.slice/crio-4e38a5af4be22289dcb733c206b2b8208723e6dc32ded83fbf0d59e9c36526dc WatchSource:0}: Error finding container 4e38a5af4be22289dcb733c206b2b8208723e6dc32ded83fbf0d59e9c36526dc: Status 404 returned error can't find the container with id 4e38a5af4be22289dcb733c206b2b8208723e6dc32ded83fbf0d59e9c36526dc
Oct 30 16:23:13.951727 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:13.951696    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66645"
Oct 30 16:23:14.145950 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.145919    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f8bbb4ff-gsfz5"]
Oct 30 16:23:14.147961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.147941    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.169960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.169929    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dmtr7\""
Oct 30 16:23:14.170119 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.170105    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 16:23:14.170216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.170203    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 16:23:14.170411 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.170388    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 16:23:14.170475 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.170463    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 16:23:14.170525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.170506    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 16:23:14.172234 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.172211    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f8bbb4ff-gsfz5"]
Oct 30 16:23:14.282393 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.282356    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-oauth-serving-cert\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.282586 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.282400    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-service-ca\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.282586 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.282428    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksbb\" (UniqueName: \"kubernetes.io/projected/08d82348-8cfb-4bff-8891-d8c79438d99f-kube-api-access-hksbb\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.282586 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.282533    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-serving-cert\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.282730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.282594    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-oauth-config\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.282730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.282675    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-console-config\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.282832 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.282743    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d82348-8cfb-4bff-8891-d8c79438d99f-tmp\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.383356    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksbb\" (UniqueName: \"kubernetes.io/projected/08d82348-8cfb-4bff-8891-d8c79438d99f-kube-api-access-hksbb\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.383410    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-serving-cert\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.383437    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-oauth-config\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.383478    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-console-config\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.383536    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d82348-8cfb-4bff-8891-d8c79438d99f-tmp\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.383564    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-oauth-serving-cert\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.383593    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-service-ca\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.384390    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-service-ca\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.386227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.385870    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-console-config\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.387630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.387147    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d82348-8cfb-4bff-8891-d8c79438d99f-tmp\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.387630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.387428    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-serving-cert\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.387630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.387591    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-oauth-serving-cert\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.388647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.388622    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-oauth-config\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.406034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.405997    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksbb\" (UniqueName: \"kubernetes.io/projected/08d82348-8cfb-4bff-8891-d8c79438d99f-kube-api-access-hksbb\") pod \"console-56f8bbb4ff-gsfz5\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") " pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.422112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.422070    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x" event={"ID":"5f171456-2c42-41ef-b11a-352f6d543e37","Type":"ContainerStarted","Data":"4e38a5af4be22289dcb733c206b2b8208723e6dc32ded83fbf0d59e9c36526dc"}
Oct 30 16:23:14.461641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.461604    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:14.768289 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:14.768261    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f8bbb4ff-gsfz5"]
Oct 30 16:23:14.774580 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:23:14.774548    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d82348_8cfb_4bff_8891_d8c79438d99f.slice/crio-4216ff7c51f5f6276ba38f9b26acd6a64599038f7b14e1e25ae6f593e8500e9e WatchSource:0}: Error finding container 4216ff7c51f5f6276ba38f9b26acd6a64599038f7b14e1e25ae6f593e8500e9e: Status 404 returned error can't find the container with id 4216ff7c51f5f6276ba38f9b26acd6a64599038f7b14e1e25ae6f593e8500e9e
Oct 30 16:23:15.433124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:15.432640    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x" event={"ID":"5f171456-2c42-41ef-b11a-352f6d543e37","Type":"ContainerStarted","Data":"effd7aac104fd309e63ca6f16269c15f56238946bc6917b51a3e7aa5da36ac0c"}
Oct 30 16:23:15.433124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:15.432725    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x" event={"ID":"5f171456-2c42-41ef-b11a-352f6d543e37","Type":"ContainerStarted","Data":"e8d9ac85cdb7985e5c1a3ab1d686f302d890b05de1f435a0854bdf1e61a0bc92"}
Oct 30 16:23:15.434297 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:15.434270    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f8bbb4ff-gsfz5" event={"ID":"08d82348-8cfb-4bff-8891-d8c79438d99f","Type":"ContainerStarted","Data":"4216ff7c51f5f6276ba38f9b26acd6a64599038f7b14e1e25ae6f593e8500e9e"}
Oct 30 16:23:15.470198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:15.469889    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8vg7x" podStartSLOduration=2.134858656 podStartE2EDuration="3.469874875s" totalImagesPullingTime="1.335016219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:15.469402439 +0000 UTC m=+25.323280703" watchObservedRunningTime="2026-10-30 16:23:15.469874875 +0000 UTC m=+25.323753118"
Oct 30 16:23:17.491542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.491505    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bl4n4"]
Oct 30 16:23:17.500273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.500242    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.550007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.549966    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 16:23:17.550007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.549966    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 16:23:17.550296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.550038    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-grp8r\""
Oct 30 16:23:17.568828 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.568794    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 16:23:17.612060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612018    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-sys\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.612060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612061    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-root\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.612321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612144    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-textfile\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.612321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612169    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/007bee35-f9a2-4595-9409-624a42b6abfa-metrics-client-ca\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.612321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612309    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-wtmp\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.612466 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612344    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.612466 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612377    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp855\" (UniqueName: \"kubernetes.io/projected/007bee35-f9a2-4595-9409-624a42b6abfa-kube-api-access-tp855\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.612555 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612502    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-tls\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.612555 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.612543    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.713853 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.713804    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-tls\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.714034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.713864    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.714034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.713901    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-sys\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.714034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.713924    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-root\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.714034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.713954    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-textfile\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.714034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.713977    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/007bee35-f9a2-4595-9409-624a42b6abfa-metrics-client-ca\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.714219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.714035    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-wtmp\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.714219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.714062    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.714219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.714092    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp855\" (UniqueName: \"kubernetes.io/projected/007bee35-f9a2-4595-9409-624a42b6abfa-kube-api-access-tp855\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.715856 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.714995    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-sys\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.715856 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.715173    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-textfile\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.715856 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.715264    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-root\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.715856 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.715380    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-wtmp\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.715856 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.715810    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/007bee35-f9a2-4595-9409-624a42b6abfa-metrics-client-ca\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.716153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.715923    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-accelerators-collector-config\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.717357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.717333    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-tls\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.718858 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.718833    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/007bee35-f9a2-4595-9409-624a42b6abfa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.735461 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.735432    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp855\" (UniqueName: \"kubernetes.io/projected/007bee35-f9a2-4595-9409-624a42b6abfa-kube-api-access-tp855\") pod \"node-exporter-bl4n4\" (UID: \"007bee35-f9a2-4595-9409-624a42b6abfa\") " pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.819241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:17.819210    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bl4n4"
Oct 30 16:23:17.833399 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:23:17.833362    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007bee35_f9a2_4595_9409_624a42b6abfa.slice/crio-b07def733d0d53891f9dcccc869a748613268c28caba3593c1e9bcd7726cad7d WatchSource:0}: Error finding container b07def733d0d53891f9dcccc869a748613268c28caba3593c1e9bcd7726cad7d: Status 404 returned error can't find the container with id b07def733d0d53891f9dcccc869a748613268c28caba3593c1e9bcd7726cad7d
Oct 30 16:23:18.448604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.448561    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bl4n4" event={"ID":"007bee35-f9a2-4595-9409-624a42b6abfa","Type":"ContainerStarted","Data":"b07def733d0d53891f9dcccc869a748613268c28caba3593c1e9bcd7726cad7d"}
Oct 30 16:23:18.456025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.455988    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f8bbb4ff-gsfz5" event={"ID":"08d82348-8cfb-4bff-8891-d8c79438d99f","Type":"ContainerStarted","Data":"c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962"}
Oct 30 16:23:18.774417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.774361    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56f8bbb4ff-gsfz5" podStartSLOduration=1.7436086240000002 podStartE2EDuration="4.774342817s" totalImagesPullingTime="3.030734193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:18.497836239 +0000 UTC m=+28.351714510" watchObservedRunningTime="2026-10-30 16:23:18.774342817 +0000 UTC m=+28.628221083"
Oct 30 16:23:18.774864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.774845    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:23:18.780144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.780108    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.792894 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.792867    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 16:23:18.793042 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.792939    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 16:23:18.793042 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.793000    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 16:23:18.793042 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.793026    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 16:23:18.797278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.796325    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 16:23:18.797278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.796471    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fhclw\""
Oct 30 16:23:18.797278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.796530    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 16:23:18.797278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.796863    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 16:23:18.797278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.797242    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 16:23:18.797583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.797494    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 16:23:18.814454 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.814419    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:23:18.926051 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926003    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926056    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-volume\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926096    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926130    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926212    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-out\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926273    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926334    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926395    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926423    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926441    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zp5\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-kube-api-access-p5zp5\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.926512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926464    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.927007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926584    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-web-config\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:18.927007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:18.926617    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027007    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027070    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027105    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-volume\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027145    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027173    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027224    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-out\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027249    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027279    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027309    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027334    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027361    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zp5\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-kube-api-access-p5zp5\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027388    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.027573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.027427    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-web-config\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.029300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.028095    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.029300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.028172    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.029300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.029240    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.030542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.030513    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.030668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.030645    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-web-config\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.031974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.031942    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.032084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.031972    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.032084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.031988    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.032455 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.032416    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.032560 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.032484    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-volume\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.033322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.033283    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-out\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.036708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.036686    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.049970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.049941    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zp5\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-kube-api-access-p5zp5\") pod \"alertmanager-main-0\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.110593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.110552    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:23:19.461781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.461693    4497 generic.go:356] "Generic (PLEG): container finished" podID="007bee35-f9a2-4595-9409-624a42b6abfa" containerID="8a4ad674b41b81364a1210d10dafdae7e4bd31d485a33609aacd87cebebe0a6f" exitCode=0
Oct 30 16:23:19.461953 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.461772    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bl4n4" event={"ID":"007bee35-f9a2-4595-9409-624a42b6abfa","Type":"ContainerDied","Data":"8a4ad674b41b81364a1210d10dafdae7e4bd31d485a33609aacd87cebebe0a6f"}
Oct 30 16:23:19.462701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.462574    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:23:19.603139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.603113    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"]
Oct 30 16:23:19.630015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.629989    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.632508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.632480    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"]
Oct 30 16:23:19.642777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.642472    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 16:23:19.642777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.642501    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 16:23:19.642777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.642566    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 16:23:19.642777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.642646    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 16:23:19.642777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.642719    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 16:23:19.642777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.642721    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cp8zn\""
Oct 30 16:23:19.642777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.642765    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fniniiqr4l9q6\""
Oct 30 16:23:19.733385 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733309    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-tls\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.733385 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733353    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.733593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733411    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.733593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733510    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-grpc-tls\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.733675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733636    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.733675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733670    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-metrics-client-ca\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.733792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733764    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5qv\" (UniqueName: \"kubernetes.io/projected/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-kube-api-access-8m5qv\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.733903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733817    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.733955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.733898    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-tmp\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.834824 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.834786    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-tmp\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.834842    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-tls\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.834871    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.834896    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.834925    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-grpc-tls\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.834962    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.834993    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-metrics-client-ca\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.835033    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5qv\" (UniqueName: \"kubernetes.io/projected/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-kube-api-access-8m5qv\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.835063    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.835266    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-tmp\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.835900 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.835878    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-metrics-client-ca\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.837988 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.837964    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.837988 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.837977    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.839135 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.839110    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-grpc-tls\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.839257 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.839113    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.839257 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.839113    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.839514 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.839492    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-secret-thanos-querier-tls\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:19.859174 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:19.859145    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5qv\" (UniqueName: \"kubernetes.io/projected/9ada3a96-8c0d-4ec0-b3f4-24821f397f3c-kube-api-access-8m5qv\") pod \"thanos-querier-fb4d4fc79-vknk2\" (UID: \"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c\") " pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:20.029905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:20.029866    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:20.380081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:20.379997    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"]
Oct 30 16:23:20.385725 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:23:20.385696    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ada3a96_8c0d_4ec0_b3f4_24821f397f3c.slice/crio-33b69bcbcaac5547a99fc5bfca06f408fc12cf75ebd6bb4288dfdec617743225 WatchSource:0}: Error finding container 33b69bcbcaac5547a99fc5bfca06f408fc12cf75ebd6bb4288dfdec617743225: Status 404 returned error can't find the container with id 33b69bcbcaac5547a99fc5bfca06f408fc12cf75ebd6bb4288dfdec617743225
Oct 30 16:23:20.468352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:20.468298    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bl4n4" event={"ID":"007bee35-f9a2-4595-9409-624a42b6abfa","Type":"ContainerStarted","Data":"9886977990eab3725e5169e7fa9ca8f918de33431c524d072c17c588ae661fce"}
Oct 30 16:23:20.468547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:20.468409    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bl4n4" event={"ID":"007bee35-f9a2-4595-9409-624a42b6abfa","Type":"ContainerStarted","Data":"0830e40b61d4586c43aa0a3c004608c5777196fba14d63dcdc8bb1e560bf5f23"}
Oct 30 16:23:20.470783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:20.470751    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2" event={"ID":"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c","Type":"ContainerStarted","Data":"33b69bcbcaac5547a99fc5bfca06f408fc12cf75ebd6bb4288dfdec617743225"}
Oct 30 16:23:20.472333 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:20.472307    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerStarted","Data":"b5995361e9db5dc5cfb01c901df0fdffec86ed40dd4447634f022cc8491ce240"}
Oct 30 16:23:20.508419 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:20.508355    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bl4n4" podStartSLOduration=2.538725941 podStartE2EDuration="3.508338733s" totalImagesPullingTime="888.330167ms" totalInitContainerRuntime="81.282625ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:20.508048857 +0000 UTC m=+30.361927140" watchObservedRunningTime="2026-10-30 16:23:20.508338733 +0000 UTC m=+30.362217001"
Oct 30 16:23:21.478585 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.478501    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerID="37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a" exitCode=0
Oct 30 16:23:21.479266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.479244    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerDied","Data":"37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a"}
Oct 30 16:23:21.899874 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.899835    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"]
Oct 30 16:23:21.920681 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.920653    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:21.927310 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:23:21.927270    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:23:21.928534 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.928497    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"]
Oct 30 16:23:21.934206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.934030    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 16:23:21.934206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.934045    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 16:23:21.934206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.934164    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 16:23:21.934426 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.934223    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2z29q\""
Oct 30 16:23:21.934426 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.934244    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4f07e71enea39\""
Oct 30 16:23:21.936612 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.936562    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 16:23:21.954087 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.954049    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fddb9fb8-e302-4783-97b3-db16eec92029-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:21.954262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.954105    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fddb9fb8-e302-4783-97b3-db16eec92029-metrics-server-audit-profiles\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:21.954262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.954215    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4s9\" (UniqueName: \"kubernetes.io/projected/fddb9fb8-e302-4783-97b3-db16eec92029-kube-api-access-2z4s9\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:21.954262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.954254    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-secret-metrics-server-client-certs\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:21.954395 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.954282    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-client-ca-bundle\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:21.954395 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.954336    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fddb9fb8-e302-4783-97b3-db16eec92029-audit-log\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:21.954466 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:21.954402    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-secret-metrics-server-tls\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.055180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.055139    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fddb9fb8-e302-4783-97b3-db16eec92029-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.055379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.055209    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fddb9fb8-e302-4783-97b3-db16eec92029-metrics-server-audit-profiles\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.055379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.055249    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4s9\" (UniqueName: \"kubernetes.io/projected/fddb9fb8-e302-4783-97b3-db16eec92029-kube-api-access-2z4s9\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.055379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.055289    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-secret-metrics-server-client-certs\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.055379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.055315    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-client-ca-bundle\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.055379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.055342    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fddb9fb8-e302-4783-97b3-db16eec92029-audit-log\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.055658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.055389    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-secret-metrics-server-tls\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.057628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.055939    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fddb9fb8-e302-4783-97b3-db16eec92029-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.057628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.056418    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fddb9fb8-e302-4783-97b3-db16eec92029-audit-log\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.057628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.056970    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fddb9fb8-e302-4783-97b3-db16eec92029-metrics-server-audit-profiles\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.058459 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.058439    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-secret-metrics-server-client-certs\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.064886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.064862    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-client-ca-bundle\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.069684 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.069658    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fddb9fb8-e302-4783-97b3-db16eec92029-secret-metrics-server-tls\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.077245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.077211    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4s9\" (UniqueName: \"kubernetes.io/projected/fddb9fb8-e302-4783-97b3-db16eec92029-kube-api-access-2z4s9\") pod \"metrics-server-67f88ddd8c-vpdnc\" (UID: \"fddb9fb8-e302-4783-97b3-db16eec92029\") " pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.249701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.249609    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"]
Oct 30 16:23:22.251794 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.251754    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:22.278987 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.278878    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"]
Oct 30 16:23:22.279073 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.279002    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:22.291703 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.291675    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 16:23:22.291878 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.291794    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w8q5j\""
Oct 30 16:23:22.358019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.357812    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/625a8fb0-a249-4439-ab60-04fa8d52b0b6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xjbdh\" (UID: \"625a8fb0-a249-4439-ab60-04fa8d52b0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:22.358019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.357855    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvm7\" (UniqueName: \"kubernetes.io/projected/625a8fb0-a249-4439-ab60-04fa8d52b0b6-kube-api-access-4lvm7\") pod \"monitoring-plugin-77f4f76594-xjbdh\" (UID: \"625a8fb0-a249-4439-ab60-04fa8d52b0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:22.458916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.458874    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/625a8fb0-a249-4439-ab60-04fa8d52b0b6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xjbdh\" (UID: \"625a8fb0-a249-4439-ab60-04fa8d52b0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:22.459094 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.458926    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvm7\" (UniqueName: \"kubernetes.io/projected/625a8fb0-a249-4439-ab60-04fa8d52b0b6-kube-api-access-4lvm7\") pod \"monitoring-plugin-77f4f76594-xjbdh\" (UID: \"625a8fb0-a249-4439-ab60-04fa8d52b0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:22.462596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.462566    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/625a8fb0-a249-4439-ab60-04fa8d52b0b6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xjbdh\" (UID: \"625a8fb0-a249-4439-ab60-04fa8d52b0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:22.480621 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.480589    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvm7\" (UniqueName: \"kubernetes.io/projected/625a8fb0-a249-4439-ab60-04fa8d52b0b6-kube-api-access-4lvm7\") pod \"monitoring-plugin-77f4f76594-xjbdh\" (UID: \"625a8fb0-a249-4439-ab60-04fa8d52b0b6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:22.604809 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.604770    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"]
Oct 30 16:23:22.641924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.641845    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:22.971630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:22.971559    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"]
Oct 30 16:23:22.977606 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:23:22.977570    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625a8fb0_a249_4439_ab60_04fa8d52b0b6.slice/crio-d606a86e04db655c8bb1b557048af60a0036111e2c0976e4314f9a1db60cf52a WatchSource:0}: Error finding container d606a86e04db655c8bb1b557048af60a0036111e2c0976e4314f9a1db60cf52a: Status 404 returned error can't find the container with id d606a86e04db655c8bb1b557048af60a0036111e2c0976e4314f9a1db60cf52a
Oct 30 16:23:23.492343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:23.492305    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh" event={"ID":"625a8fb0-a249-4439-ab60-04fa8d52b0b6","Type":"ContainerStarted","Data":"d606a86e04db655c8bb1b557048af60a0036111e2c0976e4314f9a1db60cf52a"}
Oct 30 16:23:23.493547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:23.493526    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc" event={"ID":"fddb9fb8-e302-4783-97b3-db16eec92029","Type":"ContainerStarted","Data":"4bf010690b1ec6f20ebd6e80b87cff563a055a913391b2f1aee50660e66347e8"}
Oct 30 16:23:24.421385 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.421357    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:23:24.429398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.429275    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.452863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.450573    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 16:23:24.452863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.451648    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 16:23:24.463300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.463096    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:24.463300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.463232    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:24.484876 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.484742    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.484876 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.484825    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.484876 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.484858    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.484889    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.484913    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-web-config\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.484939    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.484960    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.484984    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485006    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485035    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485081    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485105    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485128    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485156    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485201    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-config-out\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485223    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485267    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbzw\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-kube-api-access-sbbzw\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485298    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485357    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.485431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.485427    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-config\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.506544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.506499    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 16:23:24.523836 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.523418    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:24.527541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.525562    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 16:23:24.527541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.525677    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:23:24.528946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.528400    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 16:23:24.528946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.528564    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8seaoqltuecp2\""
Oct 30 16:23:24.528946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.528698    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 16:23:24.528946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.528811    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 16:23:24.528946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.528903    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 16:23:24.529278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.528967    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 16:23:24.529278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.529040    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 16:23:24.529278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.529145    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 16:23:24.529449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.529337    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mvcd\""
Oct 30 16:23:24.529501 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.529458    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 16:23:24.529628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.529581    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 16:23:24.530022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.530001    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 16:23:24.542836 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.537568    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ccccd96fc-tkbhg"]
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556519    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerStarted","Data":"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4"}
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556582    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556598    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerStarted","Data":"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e"}
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556654    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7h5xd"
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556667    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerStarted","Data":"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8"}
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556697    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerStarted","Data":"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28"}
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556729    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7h5xd" event={"ID":"21066a6d-880f-4d1d-ad2b-9c143cd219af","Type":"ContainerStarted","Data":"db898b897923f5025bad13847be8b20617ac71659f3e9ac2d50c3574f01f515a"}
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556746    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2" event={"ID":"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c","Type":"ContainerStarted","Data":"f2b4b1e96d1edf58c85f754f226578de6cf2aa7de6971b78147cbdb3f0bfc35d"}
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556781    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2" event={"ID":"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c","Type":"ContainerStarted","Data":"f37573556b293af7e12170dc7ad54b5f1144b63130a25f181599d9b262e273e5"}
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556808    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2" event={"ID":"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c","Type":"ContainerStarted","Data":"fa9141e463cdebce3d569234c26db8e03b1761db9b8228dff03694e3158db929"}
Oct 30 16:23:24.558002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.556891    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.560573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.560318    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:23:24.569765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.568034    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ccccd96fc-tkbhg"]
Oct 30 16:23:24.571583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.571513    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587360    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587406    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587435    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587482    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587513    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-oauth-config\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587540    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-trusted-ca-bundle\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587569    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-config-out\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587596    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587637    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbzw\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-kube-api-access-sbbzw\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587663    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-serving-cert\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587696    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587720    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-config\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587760    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-service-ca\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587782    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-oauth-serving-cert\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587812    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xplkh\" (UniqueName: \"kubernetes.io/projected/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-kube-api-access-xplkh\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587843    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587897    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-config\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587924    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-tmp\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.587987    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.588063    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.588097    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.588149    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.588200    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-web-config\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.588234    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.588657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.588409    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.590300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.589588    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.594555    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.594648    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.594694    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.595214    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.595625    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.596314    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.596953    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.597426    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.598687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.598670    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.600991 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.600967    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.602166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.602126    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.602571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.602544    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.603354 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.603308    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.604514 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.604489    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-config-out\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.624204 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.624158    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbzw\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-kube-api-access-sbbzw\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.645362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.642389    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.645362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.643623    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-7h5xd" podStartSLOduration=2.246509536 podStartE2EDuration="19.643607572s" totalImagesPullingTime="17.397098036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:24.642836262 +0000 UTC m=+34.496714540" watchObservedRunningTime="2026-10-30 16:23:24.643607572 +0000 UTC m=+34.497485836"
Oct 30 16:23:24.646593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.646531    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-web-config\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.647286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.647239    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.648508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.648443    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-config\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.650128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.650081    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.650788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.650739    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.662895 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.662837    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.700249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700120    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-oauth-config\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.700249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700171    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-trusted-ca-bundle\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.700249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700244    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-serving-cert\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.700497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700278    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-config\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.700497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700316    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-service-ca\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.700497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700343    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-oauth-serving-cert\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.700497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700374    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xplkh\" (UniqueName: \"kubernetes.io/projected/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-kube-api-access-xplkh\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.700497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700418    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-tmp\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.701271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.700869    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-tmp\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.703590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.702255    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-service-ca\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.703590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.702983    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-trusted-ca-bundle\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.704586 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.704540    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-oauth-serving-cert\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.706447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.706428    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-config\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.707216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.707141    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-serving-cert\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.710957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.710937    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-oauth-config\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.735546 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.726840    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xplkh\" (UniqueName: \"kubernetes.io/projected/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-kube-api-access-xplkh\") pod \"console-7ccccd96fc-tkbhg\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") " pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:24.840982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.840603    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:24.902354 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:24.901590    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:25.233614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.233416    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:23:25.296507 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.296468    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ccccd96fc-tkbhg"]
Oct 30 16:23:25.367132 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:23:25.366571    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa677e10_e6e9_45c2_bd1d_79b16ce7ec0c.slice/crio-ef1f0b582f0fb099bedd548c78c2d90dbe84efc08d2a494e0f2be158507b7a21 WatchSource:0}: Error finding container ef1f0b582f0fb099bedd548c78c2d90dbe84efc08d2a494e0f2be158507b7a21: Status 404 returned error can't find the container with id ef1f0b582f0fb099bedd548c78c2d90dbe84efc08d2a494e0f2be158507b7a21
Oct 30 16:23:25.565611 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.565487    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerStarted","Data":"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949"}
Oct 30 16:23:25.585176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.581475    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccccd96fc-tkbhg" event={"ID":"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c","Type":"ContainerStarted","Data":"0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7"}
Oct 30 16:23:25.585176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.581573    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccccd96fc-tkbhg" event={"ID":"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c","Type":"ContainerStarted","Data":"ef1f0b582f0fb099bedd548c78c2d90dbe84efc08d2a494e0f2be158507b7a21"}
Oct 30 16:23:25.592891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.592678    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerStarted","Data":"29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f"}
Oct 30 16:23:25.592891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.592822    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerStarted","Data":"0c041a27aceaec3b500cfa09327760b7c59ce6286267725dea4ea1c79afc1e07"}
Oct 30 16:23:25.600996 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.600806    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc" event={"ID":"fddb9fb8-e302-4783-97b3-db16eec92029","Type":"ContainerStarted","Data":"6430b7d5b0bbdef57430abd94ebb3c48f5d312a6223e7875949cb99a4932c302"}
Oct 30 16:23:25.665896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.664500    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc" podStartSLOduration=1.892684156 podStartE2EDuration="4.664480653s" totalImagesPullingTime="2.771796497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:25.663723432 +0000 UTC m=+35.517601699" watchObservedRunningTime="2026-10-30 16:23:25.664480653 +0000 UTC m=+35.518358929"
Oct 30 16:23:25.665896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:25.665686    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7ccccd96fc-tkbhg" podStartSLOduration=1.665674778 podStartE2EDuration="1.665674778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:25.625652683 +0000 UTC m=+35.479530948" watchObservedRunningTime="2026-10-30 16:23:25.665674778 +0000 UTC m=+35.519553071"
Oct 30 16:23:26.610393 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:26.610299    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2" event={"ID":"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c","Type":"ContainerStarted","Data":"0d0f7b70e6e365effb46d9ca88c954dcb67ec7f3a316adeccf118c4bef7271d2"}
Oct 30 16:23:26.614587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:26.613569    4497 generic.go:356] "Generic (PLEG): container finished" podID="42fb470b-b333-420a-ad38-146103dd8f98" containerID="29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f" exitCode=0
Oct 30 16:23:26.614587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:26.613641    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerDied","Data":"29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f"}
Oct 30 16:23:26.891918 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:26.891254    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-htgvf"
Oct 30 16:23:27.623686 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.623644    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2" event={"ID":"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c","Type":"ContainerStarted","Data":"dc80a7d9bd904558c4221b48b1e039cef089785b239be7de339e1329b4a94d68"}
Oct 30 16:23:27.624291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.623732    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2" event={"ID":"9ada3a96-8c0d-4ec0-b3f4-24821f397f3c","Type":"ContainerStarted","Data":"46e96a4c6c6d0774db2026158e59e373fa54a84d835e45cef28fca803eb4f3b4"}
Oct 30 16:23:27.624811 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.624788    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:27.629807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.629775    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerStarted","Data":"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359"}
Oct 30 16:23:27.632766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.632733    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh" event={"ID":"625a8fb0-a249-4439-ab60-04fa8d52b0b6","Type":"ContainerStarted","Data":"f09f326cc5de3c8701254e070b2a143cf80a668f196a7fec44c9866839a3958f"}
Oct 30 16:23:27.633057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.633010    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:27.639728 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.639700    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh"
Oct 30 16:23:27.688533 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.688440    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2" podStartSLOduration=3.200201979 podStartE2EDuration="8.688426575s" totalImagesPullingTime="5.488224596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:27.68816657 +0000 UTC m=+37.542044835" watchObservedRunningTime="2026-10-30 16:23:27.688426575 +0000 UTC m=+37.542304840"
Oct 30 16:23:27.739138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:27.739073    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.747144334 podStartE2EDuration="9.739053324s" totalImagesPullingTime="5.973696893s" totalInitContainerRuntime="18.212097ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:27.738231483 +0000 UTC m=+37.592109749" watchObservedRunningTime="2026-10-30 16:23:27.739053324 +0000 UTC m=+37.592931592"
Oct 30 16:23:29.653974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:29.653937    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-fb4d4fc79-vknk2"
Oct 30 16:23:29.707404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:29.707339    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xjbdh" podStartSLOduration=3.901147755 podStartE2EDuration="7.707320744s" totalImagesPullingTime="3.806172989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:27.777633318 +0000 UTC m=+37.631511583" watchObservedRunningTime="2026-10-30 16:23:29.707320744 +0000 UTC m=+39.561199010"
Oct 30 16:23:29.948891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:29.948856    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ccccd96fc-tkbhg"]
Oct 30 16:23:29.985698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:29.985665    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cb774c49c-6zbld"]
Oct 30 16:23:30.006206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.006141    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.039018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.038993    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cb774c49c-6zbld"]
Oct 30 16:23:30.065567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.065531    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-oauth-serving-cert\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.065739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.065573    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjdl\" (UniqueName: \"kubernetes.io/projected/1d1613c9-a0a0-43be-a476-e151bf3c6295-kube-api-access-mpjdl\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.065810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.065736    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-serving-cert\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.065810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.065777    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-oauth-config\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.065909 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.065815    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1613c9-a0a0-43be-a476-e151bf3c6295-tmp\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.065944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.065924    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-config\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.065977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.065963    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-trusted-ca-bundle\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.066022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.066005    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-service-ca\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.168680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.168654    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-config\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.168822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.168703    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-trusted-ca-bundle\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.168822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.168742    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-service-ca\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.168822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.168785    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-oauth-serving-cert\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.168822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.168808    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjdl\" (UniqueName: \"kubernetes.io/projected/1d1613c9-a0a0-43be-a476-e151bf3c6295-kube-api-access-mpjdl\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.169035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.168884    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-serving-cert\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.169035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.168912    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-oauth-config\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.169035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.168948    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1613c9-a0a0-43be-a476-e151bf3c6295-tmp\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.169553 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.169387    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1613c9-a0a0-43be-a476-e151bf3c6295-tmp\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.170202 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.170161    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-service-ca\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.170675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.170657    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-config\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.171525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.171503    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-trusted-ca-bundle\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.172322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.172272    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-oauth-serving-cert\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.175875 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.175816    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-serving-cert\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.180122 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.180101    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-oauth-config\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.202296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.201098    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjdl\" (UniqueName: \"kubernetes.io/projected/1d1613c9-a0a0-43be-a476-e151bf3c6295-kube-api-access-mpjdl\") pod \"console-5cb774c49c-6zbld\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") " pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.370027 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.369676    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:30.655239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.654961    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerStarted","Data":"eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e"}
Oct 30 16:23:30.655239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.655035    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerStarted","Data":"57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38"}
Oct 30 16:23:30.655239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.655063    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerStarted","Data":"3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd"}
Oct 30 16:23:30.655239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.655086    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerStarted","Data":"024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b"}
Oct 30 16:23:30.655239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.655121    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerStarted","Data":"11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e"}
Oct 30 16:23:30.655239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.655144    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerStarted","Data":"be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d"}
Oct 30 16:23:30.725395 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:30.725304    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cb774c49c-6zbld"]
Oct 30 16:23:30.736658 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:23:30.736619    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1613c9_a0a0_43be_a476_e151bf3c6295.slice/crio-d9095516a251c03804376352c63f68d98aafe855db8830a2d4f0c0c7b5ae624d WatchSource:0}: Error finding container d9095516a251c03804376352c63f68d98aafe855db8830a2d4f0c0c7b5ae624d: Status 404 returned error can't find the container with id d9095516a251c03804376352c63f68d98aafe855db8830a2d4f0c0c7b5ae624d
Oct 30 16:23:31.661756 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:31.661615    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb774c49c-6zbld" event={"ID":"1d1613c9-a0a0-43be-a476-e151bf3c6295","Type":"ContainerStarted","Data":"2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1"}
Oct 30 16:23:31.661756 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:31.661711    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb774c49c-6zbld" event={"ID":"1d1613c9-a0a0-43be-a476-e151bf3c6295","Type":"ContainerStarted","Data":"d9095516a251c03804376352c63f68d98aafe855db8830a2d4f0c0c7b5ae624d"}
Oct 30 16:23:31.700051 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:31.699997    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5cb774c49c-6zbld" podStartSLOduration=2.699978842 podStartE2EDuration="2.699978842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:31.699740047 +0000 UTC m=+41.553618313" watchObservedRunningTime="2026-10-30 16:23:31.699978842 +0000 UTC m=+41.553857152"
Oct 30 16:23:31.761064 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:31.760995    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.4991995510000002 podStartE2EDuration="7.76097558s" totalImagesPullingTime="4.242699372s" totalInitContainerRuntime="19.076657ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:31.759074238 +0000 UTC m=+41.612952502" watchObservedRunningTime="2026-10-30 16:23:31.76097558 +0000 UTC m=+41.614853863"
Oct 30 16:23:34.841819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:34.841733    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:23:34.903231 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:34.903180    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:40.370640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:40.370600    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:40.371278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:40.370666    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:40.375644 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:40.375617    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:40.702510 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:40.702437    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:23:40.775124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:40.775091    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f8bbb4ff-gsfz5"]
Oct 30 16:23:42.252318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:42.252277    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:42.252727 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:42.252342    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:23:51.915411 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:23:51.915367    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:23:54.972381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:54.972305    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7ccccd96fc-tkbhg" podUID="fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" containerName="console" containerID="cri-o://0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7" gracePeriod=15
Oct 30 16:23:55.244094 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.244014    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:55.341248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341211    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-oauth-serving-cert\") pod \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") "
Oct 30 16:23:55.341425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341259    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-serving-cert\") pod \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") "
Oct 30 16:23:55.341425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341282    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-trusted-ca-bundle\") pod \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") "
Oct 30 16:23:55.341425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341342    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-oauth-config\") pod \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") "
Oct 30 16:23:55.341425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341408    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-tmp\") pod \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") "
Oct 30 16:23:55.341624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341452    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-service-ca\") pod \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") "
Oct 30 16:23:55.341624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341479    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-config\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-config\") pod \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") "
Oct 30 16:23:55.341624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341512    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-kube-api-access-xplkh\" (UniqueName: \"kubernetes.io/projected/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-kube-api-access-xplkh\") pod \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\" (UID: \"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c\") "
Oct 30 16:23:55.341843 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341803    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-tmp" pod "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" (UID: "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:23:55.341958 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341825    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-trusted-ca-bundle" pod "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" (UID: "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:23:55.342081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.341914    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-oauth-serving-cert" pod "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" (UID: "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:23:55.342081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.342055    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-config" pod "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" (UID: "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:23:55.342219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.342167    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-service-ca" pod "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" (UID: "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:23:55.343633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.343605    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-oauth-config" pod "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" (UID: "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:23:55.343633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.343625    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-serving-cert" pod "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" (UID: "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:23:55.343758 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.343665    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-kube-api-access-xplkh" pod "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" (UID: "fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c"). InnerVolumeSpecName "kube-api-access-xplkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:23:55.442974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.442938    4497 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-tmp\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:23:55.442974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.442966    4497 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:23:55.442974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.442976    4497 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:23:55.443228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.442986    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xplkh\" (UniqueName: \"kubernetes.io/projected/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-kube-api-access-xplkh\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:23:55.443228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.442997    4497 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-oauth-serving-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:23:55.443228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.443011    4497 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-serving-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:23:55.443228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.443022    4497 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-trusted-ca-bundle\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:23:55.443228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.443030    4497 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c-console-oauth-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:23:55.762746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.762707    4497 generic.go:356] "Generic (PLEG): container finished" podID="fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" containerID="0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7" exitCode=0
Oct 30 16:23:55.762913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.762751    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ccccd96fc-tkbhg"
Oct 30 16:23:55.762913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.762774    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccccd96fc-tkbhg" event={"ID":"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c","Type":"ContainerDied","Data":"0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7"}
Oct 30 16:23:55.762913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.762822    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ccccd96fc-tkbhg" event={"ID":"fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c","Type":"ContainerDied","Data":"ef1f0b582f0fb099bedd548c78c2d90dbe84efc08d2a494e0f2be158507b7a21"}
Oct 30 16:23:55.762913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.762844    4497 scope.go:122] "RemoveContainer" containerID="0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7"
Oct 30 16:23:55.771390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.771373    4497 scope.go:122] "RemoveContainer" containerID="0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7"
Oct 30 16:23:55.771667 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:23:55.771651    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7\": container with ID starting with 0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7 not found: ID does not exist" containerID="0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7"
Oct 30 16:23:55.771716 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.771677    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7"} err="failed to get container status \"0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7\": rpc error: code = NotFound desc = could not find container \"0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7\": container with ID starting with 0f1c5045d3a5823d755038eba6c3f9211e39833f38650cdce09155027f91a5d7 not found: ID does not exist"
Oct 30 16:23:55.805936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.805907    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ccccd96fc-tkbhg"]
Oct 30 16:23:55.815411 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:55.815386    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ccccd96fc-tkbhg"]
Oct 30 16:23:56.296944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:23:56.296909    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" path="/var/lib/kubelet/pods/fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c/volumes"
Oct 30 16:24:02.257648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:02.257617    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:24:02.261500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:02.261478    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67f88ddd8c-vpdnc"
Oct 30 16:24:05.795131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:05.795075    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-56f8bbb4ff-gsfz5" podUID="08d82348-8cfb-4bff-8891-d8c79438d99f" containerName="console" containerID="cri-o://c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962" gracePeriod=15
Oct 30 16:24:06.085455 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.085426    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:24:06.139777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.139747    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-oauth-serving-cert\") pod \"08d82348-8cfb-4bff-8891-d8c79438d99f\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") "
Oct 30 16:24:06.139959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.139796    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d82348-8cfb-4bff-8891-d8c79438d99f-kube-api-access-hksbb\" (UniqueName: \"kubernetes.io/projected/08d82348-8cfb-4bff-8891-d8c79438d99f-kube-api-access-hksbb\") pod \"08d82348-8cfb-4bff-8891-d8c79438d99f\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") "
Oct 30 16:24:06.139959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.139835    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-console-config\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-console-config\") pod \"08d82348-8cfb-4bff-8891-d8c79438d99f\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") "
Oct 30 16:24:06.139959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.139858    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d82348-8cfb-4bff-8891-d8c79438d99f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d82348-8cfb-4bff-8891-d8c79438d99f-tmp\") pod \"08d82348-8cfb-4bff-8891-d8c79438d99f\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") "
Oct 30 16:24:06.139959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.139879    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-service-ca\") pod \"08d82348-8cfb-4bff-8891-d8c79438d99f\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") "
Oct 30 16:24:06.139959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.139917    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-oauth-config\") pod \"08d82348-8cfb-4bff-8891-d8c79438d99f\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") "
Oct 30 16:24:06.140284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.140010    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-serving-cert\") pod \"08d82348-8cfb-4bff-8891-d8c79438d99f\" (UID: \"08d82348-8cfb-4bff-8891-d8c79438d99f\") "
Oct 30 16:24:06.140284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.140154    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d82348-8cfb-4bff-8891-d8c79438d99f-tmp" pod "08d82348-8cfb-4bff-8891-d8c79438d99f" (UID: "08d82348-8cfb-4bff-8891-d8c79438d99f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:24:06.140386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.140331    4497 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08d82348-8cfb-4bff-8891-d8c79438d99f-tmp\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:06.140386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.140338    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-console-config" pod "08d82348-8cfb-4bff-8891-d8c79438d99f" (UID: "08d82348-8cfb-4bff-8891-d8c79438d99f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:06.140386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.140345    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-oauth-serving-cert" pod "08d82348-8cfb-4bff-8891-d8c79438d99f" (UID: "08d82348-8cfb-4bff-8891-d8c79438d99f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:06.140386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.140355    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-service-ca" pod "08d82348-8cfb-4bff-8891-d8c79438d99f" (UID: "08d82348-8cfb-4bff-8891-d8c79438d99f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:06.142290 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.142252    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-oauth-config" pod "08d82348-8cfb-4bff-8891-d8c79438d99f" (UID: "08d82348-8cfb-4bff-8891-d8c79438d99f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:06.142398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.142360    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d82348-8cfb-4bff-8891-d8c79438d99f-kube-api-access-hksbb" pod "08d82348-8cfb-4bff-8891-d8c79438d99f" (UID: "08d82348-8cfb-4bff-8891-d8c79438d99f"). InnerVolumeSpecName "kube-api-access-hksbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:24:06.142398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.142376    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-serving-cert" pod "08d82348-8cfb-4bff-8891-d8c79438d99f" (UID: "08d82348-8cfb-4bff-8891-d8c79438d99f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:06.241150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.241098    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hksbb\" (UniqueName: \"kubernetes.io/projected/08d82348-8cfb-4bff-8891-d8c79438d99f-kube-api-access-hksbb\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:06.241150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.241144    4497 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-console-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:06.241150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.241155    4497 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:06.241402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.241164    4497 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-oauth-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:06.241402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.241173    4497 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08d82348-8cfb-4bff-8891-d8c79438d99f-console-serving-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:06.241402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.241181    4497 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08d82348-8cfb-4bff-8891-d8c79438d99f-oauth-serving-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:06.808950 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.808912    4497 generic.go:356] "Generic (PLEG): container finished" podID="08d82348-8cfb-4bff-8891-d8c79438d99f" containerID="c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962" exitCode=0
Oct 30 16:24:06.809447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.808974    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f8bbb4ff-gsfz5"
Oct 30 16:24:06.809447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.808994    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f8bbb4ff-gsfz5" event={"ID":"08d82348-8cfb-4bff-8891-d8c79438d99f","Type":"ContainerDied","Data":"c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962"}
Oct 30 16:24:06.809447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.809068    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f8bbb4ff-gsfz5" event={"ID":"08d82348-8cfb-4bff-8891-d8c79438d99f","Type":"ContainerDied","Data":"4216ff7c51f5f6276ba38f9b26acd6a64599038f7b14e1e25ae6f593e8500e9e"}
Oct 30 16:24:06.809447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.809089    4497 scope.go:122] "RemoveContainer" containerID="c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962"
Oct 30 16:24:06.817043 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.817025    4497 scope.go:122] "RemoveContainer" containerID="c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962"
Oct 30 16:24:06.817335 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:06.817316    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962\": container with ID starting with c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962 not found: ID does not exist" containerID="c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962"
Oct 30 16:24:06.817424 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.817341    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962"} err="failed to get container status \"c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962\": rpc error: code = NotFound desc = could not find container \"c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962\": container with ID starting with c54758334c247276e3ac2b833e80938a0d7c7440ceec9ecf716d1270fb093962 not found: ID does not exist"
Oct 30 16:24:06.852263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.852228    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f8bbb4ff-gsfz5"]
Oct 30 16:24:06.862274 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:06.862250    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56f8bbb4ff-gsfz5"]
Oct 30 16:24:08.297325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:08.297294    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d82348-8cfb-4bff-8891-d8c79438d99f" path="/var/lib/kubelet/pods/08d82348-8cfb-4bff-8891-d8c79438d99f/volumes"
Oct 30 16:24:09.057949 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:09.057909    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:09.110787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:09.110763    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:09.836107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:09.836077    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:14.840218 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:14.840166    4497 generic.go:356] "Generic (PLEG): container finished" podID="c8aea925-85ca-48f2-a6a3-03eefedf6022" containerID="03e7ce2b77dd4a1a4de21b996c16a9141ee32c1e965b25f5d8f4d779d33679bd" exitCode=0
Oct 30 16:24:14.840218 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:14.840212    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jj2nm-v6brc" event={"ID":"c8aea925-85ca-48f2-a6a3-03eefedf6022","Type":"ContainerDied","Data":"03e7ce2b77dd4a1a4de21b996c16a9141ee32c1e965b25f5d8f4d779d33679bd"}
Oct 30 16:24:15.751068 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:15.751035    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jj2nm-v6brc"]
Oct 30 16:24:15.995678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:15.995644    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:24:16.131486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.131449    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle\") pod \"c8aea925-85ca-48f2-a6a3-03eefedf6022\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") "
Oct 30 16:24:16.131661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.131494    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn\" (UniqueName: \"kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn\") pod \"c8aea925-85ca-48f2-a6a3-03eefedf6022\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") "
Oct 30 16:24:16.131661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.131519    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle\") pod \"c8aea925-85ca-48f2-a6a3-03eefedf6022\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") "
Oct 30 16:24:16.131661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.131587    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path\") pod \"c8aea925-85ca-48f2-a6a3-03eefedf6022\" (UID: \"c8aea925-85ca-48f2-a6a3-03eefedf6022\") "
Oct 30 16:24:16.131966 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.131941    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle" pod "c8aea925-85ca-48f2-a6a3-03eefedf6022" (UID: "c8aea925-85ca-48f2-a6a3-03eefedf6022"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:16.132068 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.131963    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle" pod "c8aea925-85ca-48f2-a6a3-03eefedf6022" (UID: "c8aea925-85ca-48f2-a6a3-03eefedf6022"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:16.132068 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.131986    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path" pod "c8aea925-85ca-48f2-a6a3-03eefedf6022" (UID: "c8aea925-85ca-48f2-a6a3-03eefedf6022"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:24:16.133851 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.133831    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn" pod "c8aea925-85ca-48f2-a6a3-03eefedf6022" (UID: "c8aea925-85ca-48f2-a6a3-03eefedf6022"). InnerVolumeSpecName "kube-api-access-ptftn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:24:16.232162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.232124    4497 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-service-ca-bundle\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:16.232162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.232156    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptftn\" (UniqueName: \"kubernetes.io/projected/c8aea925-85ca-48f2-a6a3-03eefedf6022-kube-api-access-ptftn\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:16.232162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.232166    4497 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8aea925-85ca-48f2-a6a3-03eefedf6022-trusted-ca-bundle\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:16.232162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.232176    4497 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c8aea925-85ca-48f2-a6a3-03eefedf6022-archives-path\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:16.848412 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.848381    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jj2nm-v6brc"
Oct 30 16:24:16.848412 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.848404    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jj2nm-v6brc" event={"ID":"c8aea925-85ca-48f2-a6a3-03eefedf6022","Type":"ContainerDied","Data":"0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e"}
Oct 30 16:24:16.848652 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.848483    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cc77813161a10e1f2fc418b8e40864553c11ceb8859273762ad52bbc491e48e"
Oct 30 16:24:16.921544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.921507    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jj2nm-v6brc"]
Oct 30 16:24:16.924470 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:16.924447    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-jj2nm-v6brc"]
Oct 30 16:24:18.297004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:18.296961    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8aea925-85ca-48f2-a6a3-03eefedf6022" path="/var/lib/kubelet/pods/c8aea925-85ca-48f2-a6a3-03eefedf6022/volumes"
Oct 30 16:24:19.183158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.183125    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:24:19.183443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.183421    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="alertmanager" containerID="cri-o://3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28" gracePeriod=120
Oct 30 16:24:19.183669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.183486    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy-metric" containerID="cri-o://ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949" gracePeriod=120
Oct 30 16:24:19.183669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.183514    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy-web" containerID="cri-o://ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e" gracePeriod=120
Oct 30 16:24:19.183669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.183566    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="prom-label-proxy" containerID="cri-o://18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359" gracePeriod=120
Oct 30 16:24:19.183669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.183578    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="config-reloader" containerID="cri-o://eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8" gracePeriod=120
Oct 30 16:24:19.183669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.183530    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy" containerID="cri-o://ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4" gracePeriod=120
Oct 30 16:24:19.861754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.861725    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerID="18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359" exitCode=0
Oct 30 16:24:19.861754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.861748    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerID="ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4" exitCode=0
Oct 30 16:24:19.861754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.861755    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerID="eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8" exitCode=0
Oct 30 16:24:19.861754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.861761    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerID="3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28" exitCode=0
Oct 30 16:24:19.862243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.861800    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerDied","Data":"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359"}
Oct 30 16:24:19.862243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.861875    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerDied","Data":"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4"}
Oct 30 16:24:19.862243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.861902    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerDied","Data":"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8"}
Oct 30 16:24:19.862243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:19.861928    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerDied","Data":"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28"}
Oct 30 16:24:20.434477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.434453    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:24:20.573315 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.573279    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-out\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.573485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.573366    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-trusted-ca-bundle\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.573485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.573403    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.573485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.573439    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-metrics-client-ca\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.573485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.573467    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-kube-api-access-p5zp5\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-kube-api-access-p5zp5\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.573708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.573514    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-web\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.573849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.573808    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-trusted-ca-bundle" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:20.573946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.573924    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-metrics-client-ca" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:20.574159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574136    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-web-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-web-config\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.574381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574215    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-main-db\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.574381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574261    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-volume\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-volume\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.574381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574312    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.574381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574342    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-cluster-tls-config\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.574381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574374    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-main-tls\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.574779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574417    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-tls-assets\") pod \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\" (UID: \"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931\") "
Oct 30 16:24:20.574779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574610    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-main-db" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:24:20.574779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574719    4497 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.574779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574741    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-metrics-client-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.574779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.574757    4497 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-alertmanager-main-db\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.577605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.577578    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-kube-api-access-p5zp5" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "kube-api-access-p5zp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:24:20.578383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.578356    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-web" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:20.578482 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.578399    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-out" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:24:20.580562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.580460    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:20.580562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.580476    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-volume" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:20.580562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.580498    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-main-tls" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:20.580562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.580520    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-metric" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:20.580562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.580538    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-tls-assets" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:24:20.582571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.582553    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-cluster-tls-config" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:20.589644 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.589619    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-web-config" pod "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" (UID: "f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:20.675392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675357    4497 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675387    4497 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-web-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675398    4497 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675407    4497 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675416    4497 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-cluster-tls-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675425    4497 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-main-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675433    4497 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-tls-assets\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675441    4497 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-config-out\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675449    4497 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.675615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.675457    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5zp5\" (UniqueName: \"kubernetes.io/projected/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931-kube-api-access-p5zp5\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:20.873641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.873556    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerID="ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949" exitCode=0
Oct 30 16:24:20.873641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.873581    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerID="ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e" exitCode=0
Oct 30 16:24:20.874161 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.873642    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 16:24:20.874161 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.873638    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerDied","Data":"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949"}
Oct 30 16:24:20.874161 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.873755    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerDied","Data":"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e"}
Oct 30 16:24:20.874161 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.873805    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931","Type":"ContainerDied","Data":"b5995361e9db5dc5cfb01c901df0fdffec86ed40dd4447634f022cc8491ce240"}
Oct 30 16:24:20.874161 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.873823    4497 scope.go:122] "RemoveContainer" containerID="18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359"
Oct 30 16:24:20.882194 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.882160    4497 scope.go:122] "RemoveContainer" containerID="ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949"
Oct 30 16:24:20.888887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.888870    4497 scope.go:122] "RemoveContainer" containerID="ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4"
Oct 30 16:24:20.895819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.895804    4497 scope.go:122] "RemoveContainer" containerID="ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e"
Oct 30 16:24:20.902345 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.902319    4497 scope.go:122] "RemoveContainer" containerID="eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8"
Oct 30 16:24:20.908979 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.908962    4497 scope.go:122] "RemoveContainer" containerID="3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28"
Oct 30 16:24:20.915708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.915691    4497 scope.go:122] "RemoveContainer" containerID="37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a"
Oct 30 16:24:20.922217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.922202    4497 scope.go:122] "RemoveContainer" containerID="18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359"
Oct 30 16:24:20.922470 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:20.922451    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359\": container with ID starting with 18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359 not found: ID does not exist" containerID="18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359"
Oct 30 16:24:20.922524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.922481    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359"} err="failed to get container status \"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359\": rpc error: code = NotFound desc = could not find container \"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359\": container with ID starting with 18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359 not found: ID does not exist"
Oct 30 16:24:20.922524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.922501    4497 scope.go:122] "RemoveContainer" containerID="ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949"
Oct 30 16:24:20.922732 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:20.922714    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949\": container with ID starting with ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949 not found: ID does not exist" containerID="ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949"
Oct 30 16:24:20.922800 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.922736    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949"} err="failed to get container status \"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949\": rpc error: code = NotFound desc = could not find container \"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949\": container with ID starting with ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949 not found: ID does not exist"
Oct 30 16:24:20.922800 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.922750    4497 scope.go:122] "RemoveContainer" containerID="ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4"
Oct 30 16:24:20.923006 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:20.922990    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4\": container with ID starting with ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4 not found: ID does not exist" containerID="ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4"
Oct 30 16:24:20.923074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923007    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4"} err="failed to get container status \"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4\": rpc error: code = NotFound desc = could not find container \"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4\": container with ID starting with ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4 not found: ID does not exist"
Oct 30 16:24:20.923074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923019    4497 scope.go:122] "RemoveContainer" containerID="ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e"
Oct 30 16:24:20.923240 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:20.923225    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e\": container with ID starting with ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e not found: ID does not exist" containerID="ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e"
Oct 30 16:24:20.923284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923245    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e"} err="failed to get container status \"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e\": rpc error: code = NotFound desc = could not find container \"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e\": container with ID starting with ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e not found: ID does not exist"
Oct 30 16:24:20.923284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923257    4497 scope.go:122] "RemoveContainer" containerID="eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8"
Oct 30 16:24:20.923454 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:20.923436    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8\": container with ID starting with eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8 not found: ID does not exist" containerID="eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8"
Oct 30 16:24:20.923495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923462    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8"} err="failed to get container status \"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8\": rpc error: code = NotFound desc = could not find container \"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8\": container with ID starting with eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8 not found: ID does not exist"
Oct 30 16:24:20.923495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923478    4497 scope.go:122] "RemoveContainer" containerID="3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28"
Oct 30 16:24:20.923686 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:20.923670    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28\": container with ID starting with 3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28 not found: ID does not exist" containerID="3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28"
Oct 30 16:24:20.923724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923689    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28"} err="failed to get container status \"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28\": rpc error: code = NotFound desc = could not find container \"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28\": container with ID starting with 3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28 not found: ID does not exist"
Oct 30 16:24:20.923724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923702    4497 scope.go:122] "RemoveContainer" containerID="37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a"
Oct 30 16:24:20.923886 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:20.923871    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a\": container with ID starting with 37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a not found: ID does not exist" containerID="37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a"
Oct 30 16:24:20.923924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923889    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a"} err="failed to get container status \"37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a\": rpc error: code = NotFound desc = could not find container \"37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a\": container with ID starting with 37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a not found: ID does not exist"
Oct 30 16:24:20.923924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.923901    4497 scope.go:122] "RemoveContainer" containerID="18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359"
Oct 30 16:24:20.924129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924113    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359"} err="failed to get container status \"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359\": rpc error: code = NotFound desc = could not find container \"18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359\": container with ID starting with 18b16a64fddefa23dd00931a4d67ae6e600d44a10b80303a0f10d64f7acd4359 not found: ID does not exist"
Oct 30 16:24:20.924129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924128    4497 scope.go:122] "RemoveContainer" containerID="ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949"
Oct 30 16:24:20.924348 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924332    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949"} err="failed to get container status \"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949\": rpc error: code = NotFound desc = could not find container \"ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949\": container with ID starting with ae23fcec7037af58e5acc1a3fee49a20b1b3202781bc639f33be68510ac21949 not found: ID does not exist"
Oct 30 16:24:20.924391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924348    4497 scope.go:122] "RemoveContainer" containerID="ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4"
Oct 30 16:24:20.924534 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924518    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4"} err="failed to get container status \"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4\": rpc error: code = NotFound desc = could not find container \"ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4\": container with ID starting with ea9447d9d898cb24e9dad3f8c067b82b99ec38dcef4e691600432ff23dc383a4 not found: ID does not exist"
Oct 30 16:24:20.924574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924534    4497 scope.go:122] "RemoveContainer" containerID="ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e"
Oct 30 16:24:20.924720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924696    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e"} err="failed to get container status \"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e\": rpc error: code = NotFound desc = could not find container \"ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e\": container with ID starting with ac64cdeb775317ffc18489a22d92822f7ad1f8819d45e1419c67070830b3c95e not found: ID does not exist"
Oct 30 16:24:20.924720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924712    4497 scope.go:122] "RemoveContainer" containerID="eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8"
Oct 30 16:24:20.924894 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924879    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8"} err="failed to get container status \"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8\": rpc error: code = NotFound desc = could not find container \"eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8\": container with ID starting with eaef57dc1f14def8bfa8bf9837b6a7ea7f282202636087f05c1e30df6ceb95a8 not found: ID does not exist"
Oct 30 16:24:20.924935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.924894    4497 scope.go:122] "RemoveContainer" containerID="3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28"
Oct 30 16:24:20.925069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.925053    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28"} err="failed to get container status \"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28\": rpc error: code = NotFound desc = could not find container \"3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28\": container with ID starting with 3ee7a9ea546e8efae67cf78654e31a8e84a6eeaa6b8368885128aad42b790c28 not found: ID does not exist"
Oct 30 16:24:20.925109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.925069    4497 scope.go:122] "RemoveContainer" containerID="37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a"
Oct 30 16:24:20.925293 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.925276    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a"} err="failed to get container status \"37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a\": rpc error: code = NotFound desc = could not find container \"37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a\": container with ID starting with 37d56e825b02724289ee0adac625eb1df54da9918bda8c7604f9327721c1cf2a not found: ID does not exist"
Oct 30 16:24:20.927525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.927506    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:24:20.936840 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:20.936821    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 16:24:21.914213 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:21.914150    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:24:22.296847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:22.296807    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" path="/var/lib/kubelet/pods/f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931/volumes"
Oct 30 16:24:23.810180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:23.810108    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:24:23.810704 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:23.810539    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="prometheus" containerID="cri-o://be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d" gracePeriod=600
Oct 30 16:24:23.810704 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:23.810570    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy" containerID="cri-o://57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38" gracePeriod=600
Oct 30 16:24:23.810799 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:23.810720    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy-web" containerID="cri-o://3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd" gracePeriod=600
Oct 30 16:24:23.810799 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:23.810733    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy-thanos" containerID="cri-o://eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e" gracePeriod=600
Oct 30 16:24:23.810799 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:23.810745    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="config-reloader" containerID="cri-o://11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e" gracePeriod=600
Oct 30 16:24:23.810799 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:23.810768    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="thanos-sidecar" containerID="cri-o://024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b" gracePeriod=600
Oct 30 16:24:24.820917 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:24.820874    4497 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 16:24:24.821326 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:24.820937    4497 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 16:24:24.821326 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:24.820955    4497 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 16:24:24.821326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.820970    4497 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Oct 30 16:24:24.890799 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890771    4497 generic.go:356] "Generic (PLEG): container finished" podID="42fb470b-b333-420a-ad38-146103dd8f98" containerID="eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e" exitCode=0
Oct 30 16:24:24.890799 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890796    4497 generic.go:356] "Generic (PLEG): container finished" podID="42fb470b-b333-420a-ad38-146103dd8f98" containerID="57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38" exitCode=0
Oct 30 16:24:24.890799 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890804    4497 generic.go:356] "Generic (PLEG): container finished" podID="42fb470b-b333-420a-ad38-146103dd8f98" containerID="024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b" exitCode=0
Oct 30 16:24:24.890973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890810    4497 generic.go:356] "Generic (PLEG): container finished" podID="42fb470b-b333-420a-ad38-146103dd8f98" containerID="11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e" exitCode=0
Oct 30 16:24:24.890973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890816    4497 generic.go:356] "Generic (PLEG): container finished" podID="42fb470b-b333-420a-ad38-146103dd8f98" containerID="be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d" exitCode=0
Oct 30 16:24:24.890973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890838    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerDied","Data":"eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e"}
Oct 30 16:24:24.890973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890901    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerDied","Data":"57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38"}
Oct 30 16:24:24.890973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890928    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerDied","Data":"024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b"}
Oct 30 16:24:24.890973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890951    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerDied","Data":"11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e"}
Oct 30 16:24:24.890973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:24.890975    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerDied","Data":"be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d"}
Oct 30 16:24:25.063470 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.063448    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:25.117360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117285    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-1\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117325    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117346    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-web-config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-web-config\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117367    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-metrics-client-ca\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117397    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-metrics-client-certs\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117428    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-2\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117455    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-config-out\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-config-out\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117509    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-config\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117535    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117567    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-tls\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.117620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117597    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-serving-certs-ca-bundle\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117633    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-0\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117673    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-kube-rbac-proxy\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117708    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-db\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117722    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-1" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117737    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-grpc-tls\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117754    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-metrics-client-ca" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117794    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-tls-assets\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-tls-assets\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117846    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-thanos-prometheus-http-client-file\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117940    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-kubelet-serving-ca-bundle\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.117968    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-trusted-ca-bundle\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.118001    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-kube-api-access-sbbzw\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-kube-api-access-sbbzw\") pod \"42fb470b-b333-420a-ad38-146103dd8f98\" (UID: \"42fb470b-b333-420a-ad38-146103dd8f98\") "
Oct 30 16:24:25.118566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.118080    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-2" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:25.118566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.118106    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-serving-certs-ca-bundle" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:25.118566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.118301    4497 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.118566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.118327    4497 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.118566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.118343    4497 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-metrics-client-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.118566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.118358    4497 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.119950 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.119177    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-kubelet-serving-ca-bundle" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:25.119950 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.119843    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-trusted-ca-bundle" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:25.119950 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.119945    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-db" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:24:25.120718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.120677    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-0" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:24:25.121119 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.121092    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-config" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.121535 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.121487    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-kube-rbac-proxy-web" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.121638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.121588    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-metrics-client-certs" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.121942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.121906    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-grpc-tls" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.122023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.121946    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-thanos-prometheus-http-client-file" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.122495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.122468    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-tls" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.122930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.122906    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-kube-rbac-proxy" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.123074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.123037    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-config-out" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:24:25.123171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.123139    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-kube-api-access-sbbzw" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "kube-api-access-sbbzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:24:25.123242 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.123177    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-thanos-sidecar-tls" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.123393 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.123375    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-tls-assets" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:24:25.131573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.131554    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-web-config" pod "42fb470b-b333-420a-ad38-146103dd8f98" (UID: "42fb470b-b333-420a-ad38-146103dd8f98"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:24:25.218869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218827    4497 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.218869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218866    4497 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.218869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218878    4497 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218888    4497 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218898    4497 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-kube-rbac-proxy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218906    4497 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-k8s-db\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218914    4497 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-grpc-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218924    4497 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-tls-assets\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218931    4497 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218941    4497 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218949    4497 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42fb470b-b333-420a-ad38-146103dd8f98-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218957    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbbzw\" (UniqueName: \"kubernetes.io/projected/42fb470b-b333-420a-ad38-146103dd8f98-kube-api-access-sbbzw\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218965    4497 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218973    4497 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-web-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218981    4497 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42fb470b-b333-420a-ad38-146103dd8f98-secret-metrics-client-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.219095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.218990    4497 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42fb470b-b333-420a-ad38-146103dd8f98-config-out\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:24:25.897218 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.897167    4497 generic.go:356] "Generic (PLEG): container finished" podID="42fb470b-b333-420a-ad38-146103dd8f98" containerID="3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd" exitCode=0
Oct 30 16:24:25.897643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.897249    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerDied","Data":"3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd"}
Oct 30 16:24:25.897643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.897265    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:25.897643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.897340    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42fb470b-b333-420a-ad38-146103dd8f98","Type":"ContainerDied","Data":"0c041a27aceaec3b500cfa09327760b7c59ce6286267725dea4ea1c79afc1e07"}
Oct 30 16:24:25.897643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.897372    4497 scope.go:122] "RemoveContainer" containerID="eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e"
Oct 30 16:24:25.904887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.904865    4497 scope.go:122] "RemoveContainer" containerID="57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38"
Oct 30 16:24:25.911636 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.911612    4497 scope.go:122] "RemoveContainer" containerID="3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd"
Oct 30 16:24:25.918198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.918171    4497 scope.go:122] "RemoveContainer" containerID="024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b"
Oct 30 16:24:25.925258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.925238    4497 scope.go:122] "RemoveContainer" containerID="11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e"
Oct 30 16:24:25.931816 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.931670    4497 scope.go:122] "RemoveContainer" containerID="be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d"
Oct 30 16:24:25.938594 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.938579    4497 scope.go:122] "RemoveContainer" containerID="29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f"
Oct 30 16:24:25.944844 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.944830    4497 scope.go:122] "RemoveContainer" containerID="eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e"
Oct 30 16:24:25.945092 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:25.945073    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e\": container with ID starting with eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e not found: ID does not exist" containerID="eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e"
Oct 30 16:24:25.945150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.945101    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e"} err="failed to get container status \"eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e\": rpc error: code = NotFound desc = could not find container \"eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e\": container with ID starting with eb99fdf21e5b6956e27160ee68475422d55d5b2f5e4df4069377b9c05281279e not found: ID does not exist"
Oct 30 16:24:25.945150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.945117    4497 scope.go:122] "RemoveContainer" containerID="57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38"
Oct 30 16:24:25.945364 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:25.945348    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38\": container with ID starting with 57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38 not found: ID does not exist" containerID="57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38"
Oct 30 16:24:25.945404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.945367    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38"} err="failed to get container status \"57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38\": rpc error: code = NotFound desc = could not find container \"57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38\": container with ID starting with 57061695d17732272faaf7c0ffae57c6f1ec517fb640fd0f35859bda7790dd38 not found: ID does not exist"
Oct 30 16:24:25.945404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.945381    4497 scope.go:122] "RemoveContainer" containerID="3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd"
Oct 30 16:24:25.945615 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:25.945601    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd\": container with ID starting with 3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd not found: ID does not exist" containerID="3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd"
Oct 30 16:24:25.945659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.945617    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd"} err="failed to get container status \"3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd\": rpc error: code = NotFound desc = could not find container \"3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd\": container with ID starting with 3469d746c74402a2d17c8c165d73838a0e348895a96cb191105c0725faacb9cd not found: ID does not exist"
Oct 30 16:24:25.945659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.945636    4497 scope.go:122] "RemoveContainer" containerID="024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b"
Oct 30 16:24:25.945839 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:25.945824    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b\": container with ID starting with 024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b not found: ID does not exist" containerID="024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b"
Oct 30 16:24:25.945879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.945841    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b"} err="failed to get container status \"024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b\": rpc error: code = NotFound desc = could not find container \"024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b\": container with ID starting with 024c2e39fe2227a016e97b2f8d6dd36553fa19649a75a374e423c5c31868150b not found: ID does not exist"
Oct 30 16:24:25.945879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.945854    4497 scope.go:122] "RemoveContainer" containerID="11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e"
Oct 30 16:24:25.946041 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:25.946027    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e\": container with ID starting with 11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e not found: ID does not exist" containerID="11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e"
Oct 30 16:24:25.946080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.946044    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e"} err="failed to get container status \"11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e\": rpc error: code = NotFound desc = could not find container \"11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e\": container with ID starting with 11538db8263197295a2d86d2c321772ef90dc9414ee16b833388a40f0151ba7e not found: ID does not exist"
Oct 30 16:24:25.946080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.946055    4497 scope.go:122] "RemoveContainer" containerID="be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d"
Oct 30 16:24:25.946317 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:25.946301    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d\": container with ID starting with be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d not found: ID does not exist" containerID="be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d"
Oct 30 16:24:25.946364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.946320    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d"} err="failed to get container status \"be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d\": rpc error: code = NotFound desc = could not find container \"be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d\": container with ID starting with be052af49321ddb8e55415c2c5f5907fc5fc3458db9f8df5d346eb76c504fe1d not found: ID does not exist"
Oct 30 16:24:25.946364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.946332    4497 scope.go:122] "RemoveContainer" containerID="29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f"
Oct 30 16:24:25.946544 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:25.946524    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f\": container with ID starting with 29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f not found: ID does not exist" containerID="29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f"
Oct 30 16:24:25.946598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.946554    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f"} err="failed to get container status \"29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f\": rpc error: code = NotFound desc = could not find container \"29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f\": container with ID starting with 29cccd6e4dd9d426d50e1073a97f7e0dd43c834e4fa817f1c265c3afe51e131f not found: ID does not exist"
Oct 30 16:24:25.954246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.954223    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:24:25.963715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.963692    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:24:25.996289 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996259    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:24:25.996578 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996565    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="prom-label-proxy"
Oct 30 16:24:25.996616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996579    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="prom-label-proxy"
Oct 30 16:24:25.996616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996595    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy"
Oct 30 16:24:25.996616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996600    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy"
Oct 30 16:24:25.996616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996607    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy-web"
Oct 30 16:24:25.996616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996613    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy-web"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996620    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="alertmanager"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996625    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="alertmanager"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996632    4497 cpu_manager.go:414] "containerMap: removing container" podUID="08d82348-8cfb-4bff-8891-d8c79438d99f" containerName="console"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996637    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d82348-8cfb-4bff-8891-d8c79438d99f" containerName="console"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996643    4497 cpu_manager.go:414] "containerMap: removing container" podUID="fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" containerName="console"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996649    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" containerName="console"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996658    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="config-reloader"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996662    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="config-reloader"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996670    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="init-config-reloader"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996674    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="init-config-reloader"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996680    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c8aea925-85ca-48f2-a6a3-03eefedf6022" containerName="insights-gathering"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996684    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8aea925-85ca-48f2-a6a3-03eefedf6022" containerName="insights-gathering"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996690    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996694    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996701    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="thanos-sidecar"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996705    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="thanos-sidecar"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996712    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="prometheus"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996716    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="prometheus"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996721    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy-metric"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996725    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy-metric"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996733    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy-thanos"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996737    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy-thanos"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996747    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="init-config-reloader"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996751    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="init-config-reloader"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996757    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="config-reloader"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996761    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="config-reloader"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996766    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy-web"
Oct 30 16:24:25.996759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996770    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy-web"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996807    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="prometheus"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996815    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy-metric"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996822    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy-thanos"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996828    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="config-reloader"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996833    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy-web"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996839    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="prom-label-proxy"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996845    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="kube-rbac-proxy"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996850    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy-web"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996854    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c5e2b5-5b2d-4ada-ae38-a4d9f8c08931" containerName="alertmanager"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996859    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa677e10-e6e9-45c2-bd1d-79b16ce7ec0c" containerName="console"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996863    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d82348-8cfb-4bff-8891-d8c79438d99f" containerName="console"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996868    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="config-reloader"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996873    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8aea925-85ca-48f2-a6a3-03eefedf6022" containerName="insights-gathering"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996877    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="kube-rbac-proxy"
Oct 30 16:24:25.997493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:25.996882    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fb470b-b333-420a-ad38-146103dd8f98" containerName="thanos-sidecar"
Oct 30 16:24:26.000330 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.000311    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.013587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.013559    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 16:24:26.013764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.013737    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 16:24:26.013868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.013820    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 16:24:26.013938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.013900    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 16:24:26.014002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.013940    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 16:24:26.014098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014080    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 16:24:26.014202 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014168    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 16:24:26.014278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014244    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mvcd\""
Oct 30 16:24:26.014278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014257    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 16:24:26.014394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014285    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 16:24:26.014394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014288    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 16:24:26.014394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014309    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 16:24:26.014394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014318    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 16:24:26.014521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.014465    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8seaoqltuecp2\""
Oct 30 16:24:26.024491 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.024470    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 16:24:26.024592 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.024531    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 16:24:26.025283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025267    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025293    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025312    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025403    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025433    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-config\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025476    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025525    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/944d1bde-31c5-4c49-8d29-600c5b2feb1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025591 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025552    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025626 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025603    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025632    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025650    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025682    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfz96\" (UniqueName: \"kubernetes.io/projected/944d1bde-31c5-4c49-8d29-600c5b2feb1e-kube-api-access-qfz96\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025789 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025776    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025796    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025818    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025836    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/944d1bde-31c5-4c49-8d29-600c5b2feb1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025918 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025876    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025912    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.025980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025951    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.026013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.025983    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.033757 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.033644    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:24:26.126411 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126372    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126411 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126414    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfz96\" (UniqueName: \"kubernetes.io/projected/944d1bde-31c5-4c49-8d29-600c5b2feb1e-kube-api-access-qfz96\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126653 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126530    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126653 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126566    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126653 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126585    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126716    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/944d1bde-31c5-4c49-8d29-600c5b2feb1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126749    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126792    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126828    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126864    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126897    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.126951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126926    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126954    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.126962    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127000    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127034    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-config\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127063    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127090    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/944d1bde-31c5-4c49-8d29-600c5b2feb1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127114    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127143    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127211    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127236    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127624    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.127717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.127695    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.129464 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.129436    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.129734 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.129711    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/944d1bde-31c5-4c49-8d29-600c5b2feb1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.130052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.130026    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.130265 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.130247    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.130354 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.130268    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.130467 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.130446    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-config\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.131215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.130882    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.131215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.130979    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.131451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.131424    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.131542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.131516    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/944d1bde-31c5-4c49-8d29-600c5b2feb1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.131734 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.131708    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.132170 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.132153    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/944d1bde-31c5-4c49-8d29-600c5b2feb1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.132509 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.132489    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.132897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.132874    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.133414 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.133398    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/944d1bde-31c5-4c49-8d29-600c5b2feb1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.150877 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.150821    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfz96\" (UniqueName: \"kubernetes.io/projected/944d1bde-31c5-4c49-8d29-600c5b2feb1e-kube-api-access-qfz96\") pod \"prometheus-k8s-0\" (UID: \"944d1bde-31c5-4c49-8d29-600c5b2feb1e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.297273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.297238    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fb470b-b333-420a-ad38-146103dd8f98" path="/var/lib/kubelet/pods/42fb470b-b333-420a-ad38-146103dd8f98/volumes"
Oct 30 16:24:26.318054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.318022    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:26.655469 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.655433    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 16:24:26.660724 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:24:26.660697    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944d1bde_31c5_4c49_8d29_600c5b2feb1e.slice/crio-79007f96ae5a2221b7a5360421ebb63c636266abe18774201554f2ffca9084c5 WatchSource:0}: Error finding container 79007f96ae5a2221b7a5360421ebb63c636266abe18774201554f2ffca9084c5: Status 404 returned error can't find the container with id 79007f96ae5a2221b7a5360421ebb63c636266abe18774201554f2ffca9084c5
Oct 30 16:24:26.903355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.903320    4497 generic.go:356] "Generic (PLEG): container finished" podID="944d1bde-31c5-4c49-8d29-600c5b2feb1e" containerID="a86e0eeddccd1cffb9404139fa37734fae03d2c63f6ef375ae3948f94b9d734d" exitCode=0
Oct 30 16:24:26.903729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.903379    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"944d1bde-31c5-4c49-8d29-600c5b2feb1e","Type":"ContainerDied","Data":"a86e0eeddccd1cffb9404139fa37734fae03d2c63f6ef375ae3948f94b9d734d"}
Oct 30 16:24:26.903729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:26.903430    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"944d1bde-31c5-4c49-8d29-600c5b2feb1e","Type":"ContainerStarted","Data":"79007f96ae5a2221b7a5360421ebb63c636266abe18774201554f2ffca9084c5"}
Oct 30 16:24:27.910206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:27.910163    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"944d1bde-31c5-4c49-8d29-600c5b2feb1e","Type":"ContainerStarted","Data":"8b9604525c286ed22aa04164022557fe53040de6d79027c4c3f6bf22d33b9f1e"}
Oct 30 16:24:27.910708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:27.910259    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"944d1bde-31c5-4c49-8d29-600c5b2feb1e","Type":"ContainerStarted","Data":"f9c12ab63e5055e6f062ac6d139761ff33eed718435dcee9b372c0d76f4b0041"}
Oct 30 16:24:27.910708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:27.910299    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"944d1bde-31c5-4c49-8d29-600c5b2feb1e","Type":"ContainerStarted","Data":"79099c27c2fd6096b99962dd7127c2a3265c5ab88c5b636487bb330d6636e8ad"}
Oct 30 16:24:27.910708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:27.910337    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"944d1bde-31c5-4c49-8d29-600c5b2feb1e","Type":"ContainerStarted","Data":"fb6fdc24a95d637cc6f2abd3f3d29b7a79ca6b2626df6948a0addfd524b0d4f9"}
Oct 30 16:24:27.910708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:27.910380    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"944d1bde-31c5-4c49-8d29-600c5b2feb1e","Type":"ContainerStarted","Data":"32c69d0a495a86e9dc73c2e396aff2c93999e526ceedb0c24bfa63d3393fe1d1"}
Oct 30 16:24:27.910708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:27.910417    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"944d1bde-31c5-4c49-8d29-600c5b2feb1e","Type":"ContainerStarted","Data":"05e676c6430625a18ed3f4fb376e43ff7281a958de6ebace1f4d6e7f71879931"}
Oct 30 16:24:27.990448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:27.990394    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.968359669 podStartE2EDuration="2.990379794s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.020125ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:27.980712507 +0000 UTC m=+97.834590765" watchObservedRunningTime="2026-10-30 16:24:27.990379794 +0000 UTC m=+97.844258050"
Oct 30 16:24:31.319127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:31.319086    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:24:43.021864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.021827    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9866f47-qfc2n"]
Oct 30 16:24:43.024779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.024756    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.049748 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.049716    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9866f47-qfc2n"]
Oct 30 16:24:43.061170 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.061141    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftx4j\" (UniqueName: \"kubernetes.io/projected/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-kube-api-access-ftx4j\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.061294 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.061175    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-trusted-ca-bundle\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.061294 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.061223    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-tmp\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.061381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.061296    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-oauth-serving-cert\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.061381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.061366    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-oauth-config\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.061452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.061392    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-serving-cert\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.061452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.061440    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-config\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.061525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.061458    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-service-ca\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162117    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftx4j\" (UniqueName: \"kubernetes.io/projected/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-kube-api-access-ftx4j\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162159    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-trusted-ca-bundle\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162300    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-tmp\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162354    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-oauth-serving-cert\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162414    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-oauth-config\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162438    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-serving-cert\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162462    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-config\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162499    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-service-ca\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.162783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.162757    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-tmp\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.163105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.163081    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-oauth-serving-cert\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.163212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.163081    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-trusted-ca-bundle\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.163212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.163160    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-config\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.163212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.163168    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-service-ca\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.165362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.165341    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-serving-cert\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.165459 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.165441    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-console-oauth-config\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.184847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.184824    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftx4j\" (UniqueName: \"kubernetes.io/projected/7966f66d-bd9f-48d7-905c-10ea4bd63c0c-kube-api-access-ftx4j\") pod \"console-7c9866f47-qfc2n\" (UID: \"7966f66d-bd9f-48d7-905c-10ea4bd63c0c\") " pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.335399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.335313    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:43.632365 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.632281    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9866f47-qfc2n"]
Oct 30 16:24:43.638047 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:24:43.638025    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7966f66d_bd9f_48d7_905c_10ea4bd63c0c.slice/crio-ea4d88112f33e7f0a663534ab778fd5859c6929ae86c0bc7bb909bde9bcd49f2 WatchSource:0}: Error finding container ea4d88112f33e7f0a663534ab778fd5859c6929ae86c0bc7bb909bde9bcd49f2: Status 404 returned error can't find the container with id ea4d88112f33e7f0a663534ab778fd5859c6929ae86c0bc7bb909bde9bcd49f2
Oct 30 16:24:43.985368 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.985286    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9866f47-qfc2n" event={"ID":"7966f66d-bd9f-48d7-905c-10ea4bd63c0c","Type":"ContainerStarted","Data":"b5dcb93a977fb04c6ed07ae5cdb7df3b779f9d32927c50f0abf1315395f7e1fd"}
Oct 30 16:24:43.985368 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:43.985333    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9866f47-qfc2n" event={"ID":"7966f66d-bd9f-48d7-905c-10ea4bd63c0c","Type":"ContainerStarted","Data":"ea4d88112f33e7f0a663534ab778fd5859c6929ae86c0bc7bb909bde9bcd49f2"}
Oct 30 16:24:44.033587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:44.033540    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c9866f47-qfc2n" podStartSLOduration=1.033525531 podStartE2EDuration="1.033525531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:44.032704378 +0000 UTC m=+113.886582643" watchObservedRunningTime="2026-10-30 16:24:44.033525531 +0000 UTC m=+113.887403849"
Oct 30 16:24:51.914632 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:24:51.914595    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:24:53.335687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:53.335646    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:53.336174 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:53.335711    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:53.340569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:53.340547    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:54.029118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:54.029090    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9866f47-qfc2n"
Oct 30 16:24:54.103656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:24:54.103625    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cb774c49c-6zbld"]
Oct 30 16:25:19.123294 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.123229    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5cb774c49c-6zbld" podUID="1d1613c9-a0a0-43be-a476-e151bf3c6295" containerName="console" containerID="cri-o://2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1" gracePeriod=15
Oct 30 16:25:19.363907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.363885    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:25:19.468105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468023    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-trusted-ca-bundle\") pod \"1d1613c9-a0a0-43be-a476-e151bf3c6295\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") "
Oct 30 16:25:19.468105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468064    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-service-ca\") pod \"1d1613c9-a0a0-43be-a476-e151bf3c6295\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") "
Oct 30 16:25:19.468349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468133    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-oauth-serving-cert\") pod \"1d1613c9-a0a0-43be-a476-e151bf3c6295\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") "
Oct 30 16:25:19.468349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468168    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-oauth-config\") pod \"1d1613c9-a0a0-43be-a476-e151bf3c6295\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") "
Oct 30 16:25:19.468349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468238    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-serving-cert\") pod \"1d1613c9-a0a0-43be-a476-e151bf3c6295\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") "
Oct 30 16:25:19.468349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468290    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1613c9-a0a0-43be-a476-e151bf3c6295-kube-api-access-mpjdl\" (UniqueName: \"kubernetes.io/projected/1d1613c9-a0a0-43be-a476-e151bf3c6295-kube-api-access-mpjdl\") pod \"1d1613c9-a0a0-43be-a476-e151bf3c6295\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") "
Oct 30 16:25:19.468547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468520    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1613c9-a0a0-43be-a476-e151bf3c6295-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1613c9-a0a0-43be-a476-e151bf3c6295-tmp\") pod \"1d1613c9-a0a0-43be-a476-e151bf3c6295\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") "
Oct 30 16:25:19.468601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468556    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-service-ca" pod "1d1613c9-a0a0-43be-a476-e151bf3c6295" (UID: "1d1613c9-a0a0-43be-a476-e151bf3c6295"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:25:19.468601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468553    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-trusted-ca-bundle" pod "1d1613c9-a0a0-43be-a476-e151bf3c6295" (UID: "1d1613c9-a0a0-43be-a476-e151bf3c6295"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:25:19.468601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468582    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-config\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-config\") pod \"1d1613c9-a0a0-43be-a476-e151bf3c6295\" (UID: \"1d1613c9-a0a0-43be-a476-e151bf3c6295\") "
Oct 30 16:25:19.468721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468623    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-oauth-serving-cert" pod "1d1613c9-a0a0-43be-a476-e151bf3c6295" (UID: "1d1613c9-a0a0-43be-a476-e151bf3c6295"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:25:19.468879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468851    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1613c9-a0a0-43be-a476-e151bf3c6295-tmp" pod "1d1613c9-a0a0-43be-a476-e151bf3c6295" (UID: "1d1613c9-a0a0-43be-a476-e151bf3c6295"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:25:19.468941 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468914    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-config" pod "1d1613c9-a0a0-43be-a476-e151bf3c6295" (UID: "1d1613c9-a0a0-43be-a476-e151bf3c6295"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:25:19.469014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468943    4497 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-trusted-ca-bundle\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:25:19.469014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468961    4497 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:25:19.469014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468970    4497 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-oauth-serving-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:25:19.469014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.468979    4497 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1613c9-a0a0-43be-a476-e151bf3c6295-tmp\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:25:19.470473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.470444    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-serving-cert" pod "1d1613c9-a0a0-43be-a476-e151bf3c6295" (UID: "1d1613c9-a0a0-43be-a476-e151bf3c6295"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:25:19.470576 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.470488    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-oauth-config" pod "1d1613c9-a0a0-43be-a476-e151bf3c6295" (UID: "1d1613c9-a0a0-43be-a476-e151bf3c6295"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:25:19.470616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.470570    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1613c9-a0a0-43be-a476-e151bf3c6295-kube-api-access-mpjdl" pod "1d1613c9-a0a0-43be-a476-e151bf3c6295" (UID: "1d1613c9-a0a0-43be-a476-e151bf3c6295"). InnerVolumeSpecName "kube-api-access-mpjdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:25:19.569951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.569915    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjdl\" (UniqueName: \"kubernetes.io/projected/1d1613c9-a0a0-43be-a476-e151bf3c6295-kube-api-access-mpjdl\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:25:19.569951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.569944    4497 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:25:19.569951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.569953    4497 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-oauth-config\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:25:19.569951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:19.569963    4497 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1613c9-a0a0-43be-a476-e151bf3c6295-console-serving-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:25:20.127754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.127720    4497 generic.go:356] "Generic (PLEG): container finished" podID="1d1613c9-a0a0-43be-a476-e151bf3c6295" containerID="2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1" exitCode=0
Oct 30 16:25:20.128146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.127771    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cb774c49c-6zbld"
Oct 30 16:25:20.128146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.127806    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb774c49c-6zbld" event={"ID":"1d1613c9-a0a0-43be-a476-e151bf3c6295","Type":"ContainerDied","Data":"2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1"}
Oct 30 16:25:20.128146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.127857    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cb774c49c-6zbld" event={"ID":"1d1613c9-a0a0-43be-a476-e151bf3c6295","Type":"ContainerDied","Data":"d9095516a251c03804376352c63f68d98aafe855db8830a2d4f0c0c7b5ae624d"}
Oct 30 16:25:20.128146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.127885    4497 scope.go:122] "RemoveContainer" containerID="2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1"
Oct 30 16:25:20.136377 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.136359    4497 scope.go:122] "RemoveContainer" containerID="2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1"
Oct 30 16:25:20.136646 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:25:20.136628    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1\": container with ID starting with 2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1 not found: ID does not exist" containerID="2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1"
Oct 30 16:25:20.136682 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.136656    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1"} err="failed to get container status \"2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1\": rpc error: code = NotFound desc = could not find container \"2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1\": container with ID starting with 2b869c1c441d3d894c1456b9f2bb4b37c6869caa9410627650db877c4b95f7f1 not found: ID does not exist"
Oct 30 16:25:20.170993 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.170965    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cb774c49c-6zbld"]
Oct 30 16:25:20.180697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.180669    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cb774c49c-6zbld"]
Oct 30 16:25:20.297316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:20.297288    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1613c9-a0a0-43be-a476-e151bf3c6295" path="/var/lib/kubelet/pods/1d1613c9-a0a0-43be-a476-e151bf3c6295/volumes"
Oct 30 16:25:21.914249 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:25:21.914211    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:25:26.318450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:26.318416    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:25:26.333902 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:26.333875    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:25:27.171978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:25:27.171949    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 16:25:51.914044 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:25:51.914008    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:26:21.916021 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:26:21.915984    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:26:51.914152 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:26:51.914113    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:27:11.257885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.257851    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"]
Oct 30 16:27:11.259336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.258151    4497 cpu_manager.go:414] "containerMap: removing container" podUID="1d1613c9-a0a0-43be-a476-e151bf3c6295" containerName="console"
Oct 30 16:27:11.259336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.258162    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1613c9-a0a0-43be-a476-e151bf3c6295" containerName="console"
Oct 30 16:27:11.259336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.258235    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1613c9-a0a0-43be-a476-e151bf3c6295" containerName="console"
Oct 30 16:27:11.259848 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.259833    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.285636 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.285607    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"]
Oct 30 16:27:11.340644 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.340609    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 16:27:11.340812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.340766    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 16:27:11.340812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.340789    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 16:27:11.340936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.340848    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 16:27:11.340936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.340884    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jwzjl\""
Oct 30 16:27:11.340936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.340895    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 16:27:11.370357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.370320    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.370529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.370366    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.370529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.370384    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e36e5c86-8357-4329-9231-7992efaccd0a-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.370529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.370517    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e36e5c86-8357-4329-9231-7992efaccd0a-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.370699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.370567    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.370699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.370607    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72drz\" (UniqueName: \"kubernetes.io/projected/e36e5c86-8357-4329-9231-7992efaccd0a-kube-api-access-72drz\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.472009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.471973    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.472180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.472018    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.472180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.472036    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e36e5c86-8357-4329-9231-7992efaccd0a-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.472180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.472062    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e36e5c86-8357-4329-9231-7992efaccd0a-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.472180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.472097    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.472180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.472126    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72drz\" (UniqueName: \"kubernetes.io/projected/e36e5c86-8357-4329-9231-7992efaccd0a-kube-api-access-72drz\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.472595 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.472559    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e36e5c86-8357-4329-9231-7992efaccd0a-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.474588 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.474560    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e36e5c86-8357-4329-9231-7992efaccd0a-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.474588 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.474584    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.474725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.474636    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.474725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.474714    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e36e5c86-8357-4329-9231-7992efaccd0a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.492846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.492819    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72drz\" (UniqueName: \"kubernetes.io/projected/e36e5c86-8357-4329-9231-7992efaccd0a-kube-api-access-72drz\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq\" (UID: \"e36e5c86-8357-4329-9231-7992efaccd0a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.572924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.572831    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:11.931753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:11.931668    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"]
Oct 30 16:27:11.936637 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:27:11.936612    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36e5c86_8357_4329_9231_7992efaccd0a.slice/crio-55876959347a01338a61d704fba94a0864b8e0020605f2fd68570fcaa5ac7b1a WatchSource:0}: Error finding container 55876959347a01338a61d704fba94a0864b8e0020605f2fd68570fcaa5ac7b1a: Status 404 returned error can't find the container with id 55876959347a01338a61d704fba94a0864b8e0020605f2fd68570fcaa5ac7b1a
Oct 30 16:27:12.528023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:12.527970    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq" event={"ID":"e36e5c86-8357-4329-9231-7992efaccd0a","Type":"ContainerStarted","Data":"55876959347a01338a61d704fba94a0864b8e0020605f2fd68570fcaa5ac7b1a"}
Oct 30 16:27:15.542271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:15.542235    4497 generic.go:356] "Generic (PLEG): container finished" podID="e36e5c86-8357-4329-9231-7992efaccd0a" containerID="9d47b1cee7572824f1b229093bb8813754b28e0a97d6da6bb2f7cf1b7b35fe02" exitCode=0
Oct 30 16:27:15.542738 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:15.542322    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq" event={"ID":"e36e5c86-8357-4329-9231-7992efaccd0a","Type":"ContainerDied","Data":"9d47b1cee7572824f1b229093bb8813754b28e0a97d6da6bb2f7cf1b7b35fe02"}
Oct 30 16:27:16.547540 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:16.547501    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq" event={"ID":"e36e5c86-8357-4329-9231-7992efaccd0a","Type":"ContainerStarted","Data":"e65763cc5065ab2b41e4e600109f0375990f99e67dfdbd5b036212b97482189b"}
Oct 30 16:27:16.547926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:16.547608    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:16.633374 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:16.633318    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq" podStartSLOduration=2.095438023 podStartE2EDuration="5.633302218s" totalImagesPullingTime="3.106145486s" totalInitContainerRuntime="431.718709ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:16.622847655 +0000 UTC m=+266.476725920" watchObservedRunningTime="2026-10-30 16:27:16.633302218 +0000 UTC m=+266.487180481"
Oct 30 16:27:21.915410 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:27:21.915374    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:27:27.553386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:27.553357    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-7bgvq"
Oct 30 16:27:34.472145 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.472111    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"]
Oct 30 16:27:34.476810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.476785    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"
Oct 30 16:27:34.505408 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.505376    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"]
Oct 30 16:27:34.564145 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.564113    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aabdef4a-9f99-462b-85ff-4ee9864e5122-tmp\") pod \"jobset-operator-674cd59dc5-5drqg\" (UID: \"aabdef4a-9f99-462b-85ff-4ee9864e5122\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"
Oct 30 16:27:34.564334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.564156    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzrf6\" (UniqueName: \"kubernetes.io/projected/aabdef4a-9f99-462b-85ff-4ee9864e5122-kube-api-access-dzrf6\") pod \"jobset-operator-674cd59dc5-5drqg\" (UID: \"aabdef4a-9f99-462b-85ff-4ee9864e5122\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"
Oct 30 16:27:34.565709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.565690    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-f7spq\""
Oct 30 16:27:34.580695 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.580669    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 16:27:34.580695 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.580660    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 16:27:34.665458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.665419    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aabdef4a-9f99-462b-85ff-4ee9864e5122-tmp\") pod \"jobset-operator-674cd59dc5-5drqg\" (UID: \"aabdef4a-9f99-462b-85ff-4ee9864e5122\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"
Oct 30 16:27:34.665646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.665473    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzrf6\" (UniqueName: \"kubernetes.io/projected/aabdef4a-9f99-462b-85ff-4ee9864e5122-kube-api-access-dzrf6\") pod \"jobset-operator-674cd59dc5-5drqg\" (UID: \"aabdef4a-9f99-462b-85ff-4ee9864e5122\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"
Oct 30 16:27:34.665823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.665803    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aabdef4a-9f99-462b-85ff-4ee9864e5122-tmp\") pod \"jobset-operator-674cd59dc5-5drqg\" (UID: \"aabdef4a-9f99-462b-85ff-4ee9864e5122\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"
Oct 30 16:27:34.687328 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.687295    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzrf6\" (UniqueName: \"kubernetes.io/projected/aabdef4a-9f99-462b-85ff-4ee9864e5122-kube-api-access-dzrf6\") pod \"jobset-operator-674cd59dc5-5drqg\" (UID: \"aabdef4a-9f99-462b-85ff-4ee9864e5122\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"
Oct 30 16:27:34.799426 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:34.799387    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"
Oct 30 16:27:35.120021 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.119934    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg"]
Oct 30 16:27:35.125767 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:27:35.125684    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaabdef4a_9f99_462b_85ff_4ee9864e5122.slice/crio-1c6c05197aecbe8689aa6e9447cd0407ab32d9cb4fab2f31fb2a99fd72f0c864 WatchSource:0}: Error finding container 1c6c05197aecbe8689aa6e9447cd0407ab32d9cb4fab2f31fb2a99fd72f0c864: Status 404 returned error can't find the container with id 1c6c05197aecbe8689aa6e9447cd0407ab32d9cb4fab2f31fb2a99fd72f0c864
Oct 30 16:27:35.621926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.621892    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg" event={"ID":"aabdef4a-9f99-462b-85ff-4ee9864e5122","Type":"ContainerStarted","Data":"1c6c05197aecbe8689aa6e9447cd0407ab32d9cb4fab2f31fb2a99fd72f0c864"}
Oct 30 16:27:35.660494 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.660459    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"]
Oct 30 16:27:35.663537 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.663512    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.677112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.677085    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 16:27:35.677268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.677128    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 16:27:35.685789 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.685761    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 16:27:35.685999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.685849    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 16:27:35.686908 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.686885    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bl9vt\""
Oct 30 16:27:35.688004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.687972    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"]
Oct 30 16:27:35.774676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.774630    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea41ce5-ec7f-4bb6-a64f-458422b8566d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.774676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.774673    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea41ce5-ec7f-4bb6-a64f-458422b8566d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.774916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.774719    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8fz\" (UniqueName: \"kubernetes.io/projected/aea41ce5-ec7f-4bb6-a64f-458422b8566d-kube-api-access-cx8fz\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.875810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.875713    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea41ce5-ec7f-4bb6-a64f-458422b8566d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.875810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.875765    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea41ce5-ec7f-4bb6-a64f-458422b8566d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.875810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.875804    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8fz\" (UniqueName: \"kubernetes.io/projected/aea41ce5-ec7f-4bb6-a64f-458422b8566d-kube-api-access-cx8fz\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.876898 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.876842    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aea41ce5-ec7f-4bb6-a64f-458422b8566d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.879289 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.879257    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aea41ce5-ec7f-4bb6-a64f-458422b8566d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.906628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.906593    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8fz\" (UniqueName: \"kubernetes.io/projected/aea41ce5-ec7f-4bb6-a64f-458422b8566d-kube-api-access-cx8fz\") pod \"kuadrant-console-plugin-66c4fb575d-86rqx\" (UID: \"aea41ce5-ec7f-4bb6-a64f-458422b8566d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:35.974237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:35.974178    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"
Oct 30 16:27:36.302638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:36.302569    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx"]
Oct 30 16:27:36.308971 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:27:36.308934    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaea41ce5_ec7f_4bb6_a64f_458422b8566d.slice/crio-9ffa96a2237bc852b757c1d1de21315ebde749d8e9c102855618f00ab54254d4 WatchSource:0}: Error finding container 9ffa96a2237bc852b757c1d1de21315ebde749d8e9c102855618f00ab54254d4: Status 404 returned error can't find the container with id 9ffa96a2237bc852b757c1d1de21315ebde749d8e9c102855618f00ab54254d4
Oct 30 16:27:36.627973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:36.627890    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx" event={"ID":"aea41ce5-ec7f-4bb6-a64f-458422b8566d","Type":"ContainerStarted","Data":"9ffa96a2237bc852b757c1d1de21315ebde749d8e9c102855618f00ab54254d4"}
Oct 30 16:27:37.714768 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.714733    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"]
Oct 30 16:27:37.725235 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.725174    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"
Oct 30 16:27:37.745610 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.745567    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"]
Oct 30 16:27:37.750066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.750027    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mkkk5\""
Oct 30 16:27:37.750066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.750036    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 16:27:37.758780 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.758745    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 16:27:37.796402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.796364    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7hf\" (UniqueName: \"kubernetes.io/projected/13f7db5e-95d8-45d6-bef2-d84e9e6639d5-kube-api-access-jc7hf\") pod \"cert-manager-cainjector-5c57bbd4fc-7qstp\" (UID: \"13f7db5e-95d8-45d6-bef2-d84e9e6639d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"
Oct 30 16:27:37.796607 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.796419    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f7db5e-95d8-45d6-bef2-d84e9e6639d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7qstp\" (UID: \"13f7db5e-95d8-45d6-bef2-d84e9e6639d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"
Oct 30 16:27:37.897663 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.897625    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7hf\" (UniqueName: \"kubernetes.io/projected/13f7db5e-95d8-45d6-bef2-d84e9e6639d5-kube-api-access-jc7hf\") pod \"cert-manager-cainjector-5c57bbd4fc-7qstp\" (UID: \"13f7db5e-95d8-45d6-bef2-d84e9e6639d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"
Oct 30 16:27:37.897663 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.897669    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f7db5e-95d8-45d6-bef2-d84e9e6639d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7qstp\" (UID: \"13f7db5e-95d8-45d6-bef2-d84e9e6639d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"
Oct 30 16:27:37.923368 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.923319    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7hf\" (UniqueName: \"kubernetes.io/projected/13f7db5e-95d8-45d6-bef2-d84e9e6639d5-kube-api-access-jc7hf\") pod \"cert-manager-cainjector-5c57bbd4fc-7qstp\" (UID: \"13f7db5e-95d8-45d6-bef2-d84e9e6639d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"
Oct 30 16:27:37.924982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:37.924953    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f7db5e-95d8-45d6-bef2-d84e9e6639d5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7qstp\" (UID: \"13f7db5e-95d8-45d6-bef2-d84e9e6639d5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"
Oct 30 16:27:38.040079 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:38.040052    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"
Oct 30 16:27:38.355380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:38.355294    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp"]
Oct 30 16:27:38.362989 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:27:38.362956    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13f7db5e_95d8_45d6_bef2_d84e9e6639d5.slice/crio-475d9cee3042647e94babc5f832e3475772d7c4337f0e930a100c295bace63eb WatchSource:0}: Error finding container 475d9cee3042647e94babc5f832e3475772d7c4337f0e930a100c295bace63eb: Status 404 returned error can't find the container with id 475d9cee3042647e94babc5f832e3475772d7c4337f0e930a100c295bace63eb
Oct 30 16:27:38.639868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:38.639786    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp" event={"ID":"13f7db5e-95d8-45d6-bef2-d84e9e6639d5","Type":"ContainerStarted","Data":"475d9cee3042647e94babc5f832e3475772d7c4337f0e930a100c295bace63eb"}
Oct 30 16:27:38.642118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:38.642079    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg" event={"ID":"aabdef4a-9f99-462b-85ff-4ee9864e5122","Type":"ContainerStarted","Data":"88251e367e6832790912f9b0067fa58a75ebaa57525e5e3353a66e9cce8563c4"}
Oct 30 16:27:38.681230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:38.681153    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-5drqg" podStartSLOduration=1.7745921770000002 podStartE2EDuration="4.681137332s" totalImagesPullingTime="2.906545155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:38.68080103 +0000 UTC m=+288.534679313" watchObservedRunningTime="2026-10-30 16:27:38.681137332 +0000 UTC m=+288.535015597"
Oct 30 16:27:41.659935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:41.659896    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx" event={"ID":"aea41ce5-ec7f-4bb6-a64f-458422b8566d","Type":"ContainerStarted","Data":"ec4a466fdb2b11005d5c0705c2562092abebc22a6e1d4be8945ecb8d0f1dfd05"}
Oct 30 16:27:41.695383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:41.695328    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-86rqx" podStartSLOduration=1.510338884 podStartE2EDuration="6.695309371s" totalImagesPullingTime="5.184970487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:41.6942243 +0000 UTC m=+291.548102628" watchObservedRunningTime="2026-10-30 16:27:41.695309371 +0000 UTC m=+291.549187667"
Oct 30 16:27:42.665631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:42.665596    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp" event={"ID":"13f7db5e-95d8-45d6-bef2-d84e9e6639d5","Type":"ContainerStarted","Data":"5bc3470eb607b5332dd6e94e0240d5ebb955ebb584a6aff220f61744d696a51a"}
Oct 30 16:27:42.700795 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:42.700739    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7qstp" podStartSLOduration=2.144578595 podStartE2EDuration="5.700725428s" totalImagesPullingTime="3.556146833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:42.699630921 +0000 UTC m=+292.553509185" watchObservedRunningTime="2026-10-30 16:27:42.700725428 +0000 UTC m=+292.554603693"
Oct 30 16:27:42.902309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:42.902271    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"]
Oct 30 16:27:42.905814 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:42.905792    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:42.920285 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:42.920233    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8428q\""
Oct 30 16:27:42.925956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:42.925934    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"]
Oct 30 16:27:42.945748 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:42.945722    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a524c7d-3802-443c-a1e0-056bc0e34af4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-p8zlx\" (UID: \"2a524c7d-3802-443c-a1e0-056bc0e34af4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:42.945885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:42.945842    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5f9\" (UniqueName: \"kubernetes.io/projected/2a524c7d-3802-443c-a1e0-056bc0e34af4-kube-api-access-cz5f9\") pod \"cert-manager-webhook-5dbbbf75cc-p8zlx\" (UID: \"2a524c7d-3802-443c-a1e0-056bc0e34af4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:43.046790 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.046753    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a524c7d-3802-443c-a1e0-056bc0e34af4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-p8zlx\" (UID: \"2a524c7d-3802-443c-a1e0-056bc0e34af4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:43.046946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.046822    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5f9\" (UniqueName: \"kubernetes.io/projected/2a524c7d-3802-443c-a1e0-056bc0e34af4-kube-api-access-cz5f9\") pod \"cert-manager-webhook-5dbbbf75cc-p8zlx\" (UID: \"2a524c7d-3802-443c-a1e0-056bc0e34af4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:43.073304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.073271    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a524c7d-3802-443c-a1e0-056bc0e34af4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-p8zlx\" (UID: \"2a524c7d-3802-443c-a1e0-056bc0e34af4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:43.073446 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.073401    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5f9\" (UniqueName: \"kubernetes.io/projected/2a524c7d-3802-443c-a1e0-056bc0e34af4-kube-api-access-cz5f9\") pod \"cert-manager-webhook-5dbbbf75cc-p8zlx\" (UID: \"2a524c7d-3802-443c-a1e0-056bc0e34af4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:43.216891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.216799    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:43.515378 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.515342    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"]
Oct 30 16:27:43.520610 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:27:43.520578    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a524c7d_3802_443c_a1e0_056bc0e34af4.slice/crio-08b4f275bd2f974666db67d7b82814ee314185d6c43e4d522addc269abd218ac WatchSource:0}: Error finding container 08b4f275bd2f974666db67d7b82814ee314185d6c43e4d522addc269abd218ac: Status 404 returned error can't find the container with id 08b4f275bd2f974666db67d7b82814ee314185d6c43e4d522addc269abd218ac
Oct 30 16:27:43.670818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.670786    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx" event={"ID":"2a524c7d-3802-443c-a1e0-056bc0e34af4","Type":"ContainerStarted","Data":"31c426b181d4624ebe9aadbfe06d27c8c24164077b69797b19f5a920813aedf8"}
Oct 30 16:27:43.671210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.670833    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx" event={"ID":"2a524c7d-3802-443c-a1e0-056bc0e34af4","Type":"ContainerStarted","Data":"08b4f275bd2f974666db67d7b82814ee314185d6c43e4d522addc269abd218ac"}
Oct 30 16:27:43.710536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:43.710481    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx" podStartSLOduration=1.710468141 podStartE2EDuration="1.710468141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:43.709814849 +0000 UTC m=+293.563693115" watchObservedRunningTime="2026-10-30 16:27:43.710468141 +0000 UTC m=+293.564346405"
Oct 30 16:27:44.674630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:44.674601    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:46.427698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.427666    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tpj42"]
Oct 30 16:27:46.431141 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.431113    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tpj42"
Oct 30 16:27:46.447581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.447558    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-24dsm\""
Oct 30 16:27:46.455200 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.455168    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tpj42"]
Oct 30 16:27:46.478821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.478793    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktwl2\" (UniqueName: \"kubernetes.io/projected/38823750-c0cf-4723-b475-de6e8719f3dc-kube-api-access-ktwl2\") pod \"cert-manager-597b95cb84-tpj42\" (UID: \"38823750-c0cf-4723-b475-de6e8719f3dc\") " pod="cert-manager/cert-manager-597b95cb84-tpj42"
Oct 30 16:27:46.478977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.478829    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38823750-c0cf-4723-b475-de6e8719f3dc-bound-sa-token\") pod \"cert-manager-597b95cb84-tpj42\" (UID: \"38823750-c0cf-4723-b475-de6e8719f3dc\") " pod="cert-manager/cert-manager-597b95cb84-tpj42"
Oct 30 16:27:46.580292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.580254    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwl2\" (UniqueName: \"kubernetes.io/projected/38823750-c0cf-4723-b475-de6e8719f3dc-kube-api-access-ktwl2\") pod \"cert-manager-597b95cb84-tpj42\" (UID: \"38823750-c0cf-4723-b475-de6e8719f3dc\") " pod="cert-manager/cert-manager-597b95cb84-tpj42"
Oct 30 16:27:46.580476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.580298    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38823750-c0cf-4723-b475-de6e8719f3dc-bound-sa-token\") pod \"cert-manager-597b95cb84-tpj42\" (UID: \"38823750-c0cf-4723-b475-de6e8719f3dc\") " pod="cert-manager/cert-manager-597b95cb84-tpj42"
Oct 30 16:27:46.601297 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.601262    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38823750-c0cf-4723-b475-de6e8719f3dc-bound-sa-token\") pod \"cert-manager-597b95cb84-tpj42\" (UID: \"38823750-c0cf-4723-b475-de6e8719f3dc\") " pod="cert-manager/cert-manager-597b95cb84-tpj42"
Oct 30 16:27:46.601449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.601324    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktwl2\" (UniqueName: \"kubernetes.io/projected/38823750-c0cf-4723-b475-de6e8719f3dc-kube-api-access-ktwl2\") pod \"cert-manager-597b95cb84-tpj42\" (UID: \"38823750-c0cf-4723-b475-de6e8719f3dc\") " pod="cert-manager/cert-manager-597b95cb84-tpj42"
Oct 30 16:27:46.743480 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.743398    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tpj42"
Oct 30 16:27:46.747812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.747784    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"]
Oct 30 16:27:46.751246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.751226    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:46.764294 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.764261    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 16:27:46.764420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.764399    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 16:27:46.775703 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.775676    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"]
Oct 30 16:27:46.792748 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.792718    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 16:27:46.793092 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.792943    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 16:27:46.793092 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.793004    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p6cx2\""
Oct 30 16:27:46.883126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.883093    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/941eec69-d000-41b3-9250-3ce425d3f9e7-tmp\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:46.883126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.883133    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p28r6\" (UniqueName: \"kubernetes.io/projected/941eec69-d000-41b3-9250-3ce425d3f9e7-kube-api-access-p28r6\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:46.883349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.883246    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941eec69-d000-41b3-9250-3ce425d3f9e7-tls-certs\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:46.984037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.984005    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941eec69-d000-41b3-9250-3ce425d3f9e7-tls-certs\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:46.984232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.984085    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/941eec69-d000-41b3-9250-3ce425d3f9e7-tmp\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:46.984232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.984117    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p28r6\" (UniqueName: \"kubernetes.io/projected/941eec69-d000-41b3-9250-3ce425d3f9e7-kube-api-access-p28r6\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:46.986483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.986446    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/941eec69-d000-41b3-9250-3ce425d3f9e7-tmp\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:46.986623 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:46.986600    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941eec69-d000-41b3-9250-3ce425d3f9e7-tls-certs\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:47.005136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:47.005109    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p28r6\" (UniqueName: \"kubernetes.io/projected/941eec69-d000-41b3-9250-3ce425d3f9e7-kube-api-access-p28r6\") pod \"kube-auth-proxy-b54f85fcc-bz58x\" (UID: \"941eec69-d000-41b3-9250-3ce425d3f9e7\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:47.057546 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:47.057513    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tpj42"]
Oct 30 16:27:47.063939 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:27:47.063911    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38823750_c0cf_4723_b475_de6e8719f3dc.slice/crio-0b49e60051c501872e51067ca0b834dd2dd7add9bc65540578a821f3e02aa40a WatchSource:0}: Error finding container 0b49e60051c501872e51067ca0b834dd2dd7add9bc65540578a821f3e02aa40a: Status 404 returned error can't find the container with id 0b49e60051c501872e51067ca0b834dd2dd7add9bc65540578a821f3e02aa40a
Oct 30 16:27:47.078529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:47.078503    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"
Oct 30 16:27:47.436214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:47.436109    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x"]
Oct 30 16:27:47.442954 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:27:47.442917    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941eec69_d000_41b3_9250_3ce425d3f9e7.slice/crio-52cee32dda949074b2dad492e48fb292d380fc62fc36fac614e206cef59099aa WatchSource:0}: Error finding container 52cee32dda949074b2dad492e48fb292d380fc62fc36fac614e206cef59099aa: Status 404 returned error can't find the container with id 52cee32dda949074b2dad492e48fb292d380fc62fc36fac614e206cef59099aa
Oct 30 16:27:47.689548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:47.689458    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x" event={"ID":"941eec69-d000-41b3-9250-3ce425d3f9e7","Type":"ContainerStarted","Data":"52cee32dda949074b2dad492e48fb292d380fc62fc36fac614e206cef59099aa"}
Oct 30 16:27:47.691130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:47.691094    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tpj42" event={"ID":"38823750-c0cf-4723-b475-de6e8719f3dc","Type":"ContainerStarted","Data":"6a16768f6b85a71ffc3e04995d91ab8e5dfbe9820e7a7725bb9c7f0009c3d2d0"}
Oct 30 16:27:47.691130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:47.691128    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tpj42" event={"ID":"38823750-c0cf-4723-b475-de6e8719f3dc","Type":"ContainerStarted","Data":"0b49e60051c501872e51067ca0b834dd2dd7add9bc65540578a821f3e02aa40a"}
Oct 30 16:27:47.727572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:47.727520    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tpj42" podStartSLOduration=1.727506139 podStartE2EDuration="1.727506139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:47.726426937 +0000 UTC m=+297.580305202" watchObservedRunningTime="2026-10-30 16:27:47.727506139 +0000 UTC m=+297.581384401"
Oct 30 16:27:50.680479 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:50.680447    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx"
Oct 30 16:27:50.707963 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:50.707927    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x" event={"ID":"941eec69-d000-41b3-9250-3ce425d3f9e7","Type":"ContainerStarted","Data":"04cb449bc8beaa9a4e16eaf7926c9e553c0d5e113990f7bdc3d7843793f28373"}
Oct 30 16:27:50.756933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:50.756879    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-bz58x" podStartSLOduration=2.308145253 podStartE2EDuration="4.756861951s" totalImagesPullingTime="2.448716698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:50.755599998 +0000 UTC m=+300.609478269" watchObservedRunningTime="2026-10-30 16:27:50.756861951 +0000 UTC m=+300.610740216"
Oct 30 16:27:51.914682 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:27:51.914646    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:27:55.658132 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.658098    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7m67l"]
Oct 30 16:27:55.663215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.663166    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:27:55.675357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.675334    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 16:27:55.682654 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.682629    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7m67l"]
Oct 30 16:27:55.709753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.709716    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djc9p\" (UniqueName: \"kubernetes.io/projected/5090913e-fec3-4314-8c23-e9328a4faed5-kube-api-access-djc9p\") pod \"limitador-limitador-74dfcfb894-7m67l\" (UID: \"5090913e-fec3-4314-8c23-e9328a4faed5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:27:55.709897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.709816    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5090913e-fec3-4314-8c23-e9328a4faed5-config-file\") pod \"limitador-limitador-74dfcfb894-7m67l\" (UID: \"5090913e-fec3-4314-8c23-e9328a4faed5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:27:55.798221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.798171    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7m67l"]
Oct 30 16:27:55.810271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.810240    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djc9p\" (UniqueName: \"kubernetes.io/projected/5090913e-fec3-4314-8c23-e9328a4faed5-kube-api-access-djc9p\") pod \"limitador-limitador-74dfcfb894-7m67l\" (UID: \"5090913e-fec3-4314-8c23-e9328a4faed5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:27:55.810449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.810290    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5090913e-fec3-4314-8c23-e9328a4faed5-config-file\") pod \"limitador-limitador-74dfcfb894-7m67l\" (UID: \"5090913e-fec3-4314-8c23-e9328a4faed5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:27:55.810932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.810914    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5090913e-fec3-4314-8c23-e9328a4faed5-config-file\") pod \"limitador-limitador-74dfcfb894-7m67l\" (UID: \"5090913e-fec3-4314-8c23-e9328a4faed5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:27:55.831436 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.831412    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djc9p\" (UniqueName: \"kubernetes.io/projected/5090913e-fec3-4314-8c23-e9328a4faed5-kube-api-access-djc9p\") pod \"limitador-limitador-74dfcfb894-7m67l\" (UID: \"5090913e-fec3-4314-8c23-e9328a4faed5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:27:55.975509 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:55.975418    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:27:56.276538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:56.276500    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7m67l"]
Oct 30 16:27:56.356497 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:27:56.356448    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5090913e_fec3_4314_8c23_e9328a4faed5.slice/crio-acbb05b21f14a50846d89aebd27398f014412c67fecee4e3c0fef15c5d24819e WatchSource:0}: Error finding container acbb05b21f14a50846d89aebd27398f014412c67fecee4e3c0fef15c5d24819e: Status 404 returned error can't find the container with id acbb05b21f14a50846d89aebd27398f014412c67fecee4e3c0fef15c5d24819e
Oct 30 16:27:56.358470 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:56.358454    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:27:56.733643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:27:56.733555    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l" event={"ID":"5090913e-fec3-4314-8c23-e9328a4faed5","Type":"ContainerStarted","Data":"acbb05b21f14a50846d89aebd27398f014412c67fecee4e3c0fef15c5d24819e"}
Oct 30 16:28:03.764775 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:03.764741    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l" event={"ID":"5090913e-fec3-4314-8c23-e9328a4faed5","Type":"ContainerStarted","Data":"2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881"}
Oct 30 16:28:03.765173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:03.764860    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:28:03.800923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:03.800875    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l" podStartSLOduration=2.458753453 podStartE2EDuration="8.80086285s" totalImagesPullingTime="6.342109397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:03.800857396 +0000 UTC m=+313.654735661" watchObservedRunningTime="2026-10-30 16:28:03.80086285 +0000 UTC m=+313.654741114"
Oct 30 16:28:14.772788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:14.772757    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:28:21.915665 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:28:21.915626    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:28:29.647396 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.647353    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"]
Oct 30 16:28:29.680330 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.680297    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"]
Oct 30 16:28:29.680528 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.680416    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.692757 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.692730    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 16:28:29.702350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.702323    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-tl5xr\""
Oct 30 16:28:29.702503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.702371    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 16:28:29.702503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.702396    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 16:28:29.803716 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.803680    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9f5eb82-fe35-44de-b048-06bf9a165245-cert\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.803904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.803740    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c9f5eb82-fe35-44de-b048-06bf9a165245-manager-config\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.803904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.803763    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mqh\" (UniqueName: \"kubernetes.io/projected/c9f5eb82-fe35-44de-b048-06bf9a165245-kube-api-access-66mqh\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.803904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.803894    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9f5eb82-fe35-44de-b048-06bf9a165245-metrics-certs\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.905254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.905134    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9f5eb82-fe35-44de-b048-06bf9a165245-cert\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.905254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.905230    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c9f5eb82-fe35-44de-b048-06bf9a165245-manager-config\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.905254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.905248    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mqh\" (UniqueName: \"kubernetes.io/projected/c9f5eb82-fe35-44de-b048-06bf9a165245-kube-api-access-66mqh\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.905532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.905274    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9f5eb82-fe35-44de-b048-06bf9a165245-metrics-certs\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.905965 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.905939    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c9f5eb82-fe35-44de-b048-06bf9a165245-manager-config\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.907727 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.907706    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9f5eb82-fe35-44de-b048-06bf9a165245-metrics-certs\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.907825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.907749    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9f5eb82-fe35-44de-b048-06bf9a165245-cert\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.926777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.926741    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mqh\" (UniqueName: \"kubernetes.io/projected/c9f5eb82-fe35-44de-b048-06bf9a165245-kube-api-access-66mqh\") pod \"jobset-controller-manager-fdf9b4bc5-25hrm\" (UID: \"c9f5eb82-fe35-44de-b048-06bf9a165245\") " pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:29.992618 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:29.992585    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:30.298365 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:30.298337    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"]
Oct 30 16:28:30.303622 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:28:30.303594    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f5eb82_fe35_44de_b048_06bf9a165245.slice/crio-141670b61d8b08b74b989d1ff5872bacc67cd3184588389651ea6a005645d181 WatchSource:0}: Error finding container 141670b61d8b08b74b989d1ff5872bacc67cd3184588389651ea6a005645d181: Status 404 returned error can't find the container with id 141670b61d8b08b74b989d1ff5872bacc67cd3184588389651ea6a005645d181
Oct 30 16:28:30.612599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:30.612513    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7m67l"]
Oct 30 16:28:30.612776 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:30.612737    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l" podUID="5090913e-fec3-4314-8c23-e9328a4faed5" containerName="limitador" containerID="cri-o://2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881" gracePeriod=30
Oct 30 16:28:30.881436 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:30.881355    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm" event={"ID":"c9f5eb82-fe35-44de-b048-06bf9a165245","Type":"ContainerStarted","Data":"141670b61d8b08b74b989d1ff5872bacc67cd3184588389651ea6a005645d181"}
Oct 30 16:28:31.185839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.185800    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:28:31.320977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.320295    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5090913e-fec3-4314-8c23-e9328a4faed5-config-file\" (UniqueName: \"kubernetes.io/configmap/5090913e-fec3-4314-8c23-e9328a4faed5-config-file\") pod \"5090913e-fec3-4314-8c23-e9328a4faed5\" (UID: \"5090913e-fec3-4314-8c23-e9328a4faed5\") "
Oct 30 16:28:31.320977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.320391    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5090913e-fec3-4314-8c23-e9328a4faed5-kube-api-access-djc9p\" (UniqueName: \"kubernetes.io/projected/5090913e-fec3-4314-8c23-e9328a4faed5-kube-api-access-djc9p\") pod \"5090913e-fec3-4314-8c23-e9328a4faed5\" (UID: \"5090913e-fec3-4314-8c23-e9328a4faed5\") "
Oct 30 16:28:31.320977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.320727    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5090913e-fec3-4314-8c23-e9328a4faed5-config-file" pod "5090913e-fec3-4314-8c23-e9328a4faed5" (UID: "5090913e-fec3-4314-8c23-e9328a4faed5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:28:31.323209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.323157    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5090913e-fec3-4314-8c23-e9328a4faed5-kube-api-access-djc9p" pod "5090913e-fec3-4314-8c23-e9328a4faed5" (UID: "5090913e-fec3-4314-8c23-e9328a4faed5"). InnerVolumeSpecName "kube-api-access-djc9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:28:31.422225 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.422117    4497 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5090913e-fec3-4314-8c23-e9328a4faed5-config-file\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:28:31.422225 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.422159    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djc9p\" (UniqueName: \"kubernetes.io/projected/5090913e-fec3-4314-8c23-e9328a4faed5-kube-api-access-djc9p\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:28:31.887162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.887125    4497 generic.go:356] "Generic (PLEG): container finished" podID="5090913e-fec3-4314-8c23-e9328a4faed5" containerID="2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881" exitCode=0
Oct 30 16:28:31.887627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.887206    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l"
Oct 30 16:28:31.887627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.887214    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l" event={"ID":"5090913e-fec3-4314-8c23-e9328a4faed5","Type":"ContainerDied","Data":"2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881"}
Oct 30 16:28:31.887627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.887268    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7m67l" event={"ID":"5090913e-fec3-4314-8c23-e9328a4faed5","Type":"ContainerDied","Data":"acbb05b21f14a50846d89aebd27398f014412c67fecee4e3c0fef15c5d24819e"}
Oct 30 16:28:31.887627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.887298    4497 scope.go:122] "RemoveContainer" containerID="2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881"
Oct 30 16:28:31.896040 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.895946    4497 scope.go:122] "RemoveContainer" containerID="2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881"
Oct 30 16:28:31.896397 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:28:31.896371    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881\": container with ID starting with 2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881 not found: ID does not exist" containerID="2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881"
Oct 30 16:28:31.896508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.896405    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881"} err="failed to get container status \"2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881\": rpc error: code = NotFound desc = could not find container \"2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881\": container with ID starting with 2dd05e90c420b68b293db7d32d04baa403c930d3d0e3b6eb46ec7dd9b6704881 not found: ID does not exist"
Oct 30 16:28:31.930698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.930671    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7m67l"]
Oct 30 16:28:31.940523 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:31.940491    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7m67l"]
Oct 30 16:28:32.299502 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:32.299471    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5090913e-fec3-4314-8c23-e9328a4faed5" path="/var/lib/kubelet/pods/5090913e-fec3-4314-8c23-e9328a4faed5/volumes"
Oct 30 16:28:33.899798 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:33.899761    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm" event={"ID":"c9f5eb82-fe35-44de-b048-06bf9a165245","Type":"ContainerStarted","Data":"9d6f632e8c82a3123e045caccd11f8d2d73a3f0084c0bae8752ad14559b3c154"}
Oct 30 16:28:33.900215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:33.899845    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:33.937743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:33.937694    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm" podStartSLOduration=2.158094185 podStartE2EDuration="4.937680919s" totalImagesPullingTime="2.779586734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:33.937047771 +0000 UTC m=+343.790926035" watchObservedRunningTime="2026-10-30 16:28:33.937680919 +0000 UTC m=+343.791559183"
Oct 30 16:28:44.908613 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:44.908581    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-fdf9b4bc5-25hrm"
Oct 30 16:28:46.922676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.922643    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gbc6r"]
Oct 30 16:28:46.923077 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.923007    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5090913e-fec3-4314-8c23-e9328a4faed5" containerName="limitador"
Oct 30 16:28:46.923077 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.923019    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5090913e-fec3-4314-8c23-e9328a4faed5" containerName="limitador"
Oct 30 16:28:46.923150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.923086    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5090913e-fec3-4314-8c23-e9328a4faed5" containerName="limitador"
Oct 30 16:28:46.927948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.927926    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:46.946469 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.946445    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 16:28:46.951831 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.951810    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 16:28:46.951936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.951857    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 16:28:46.952298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.952283    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jt87k\""
Oct 30 16:28:46.957557 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.957534    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gbc6r"]
Oct 30 16:28:46.964343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.964316    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b467741e-e532-40ad-9261-db4f8b0cac24-tls-certs\") pod \"model-serving-api-75bd89fdfc-gbc6r\" (UID: \"b467741e-e532-40ad-9261-db4f8b0cac24\") " pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:46.964500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:46.964479    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn26m\" (UniqueName: \"kubernetes.io/projected/b467741e-e532-40ad-9261-db4f8b0cac24-kube-api-access-pn26m\") pod \"model-serving-api-75bd89fdfc-gbc6r\" (UID: \"b467741e-e532-40ad-9261-db4f8b0cac24\") " pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:47.064858 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:47.064826    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b467741e-e532-40ad-9261-db4f8b0cac24-tls-certs\") pod \"model-serving-api-75bd89fdfc-gbc6r\" (UID: \"b467741e-e532-40ad-9261-db4f8b0cac24\") " pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:47.065043 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:47.064921    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn26m\" (UniqueName: \"kubernetes.io/projected/b467741e-e532-40ad-9261-db4f8b0cac24-kube-api-access-pn26m\") pod \"model-serving-api-75bd89fdfc-gbc6r\" (UID: \"b467741e-e532-40ad-9261-db4f8b0cac24\") " pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:47.067456 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:47.067429    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b467741e-e532-40ad-9261-db4f8b0cac24-tls-certs\") pod \"model-serving-api-75bd89fdfc-gbc6r\" (UID: \"b467741e-e532-40ad-9261-db4f8b0cac24\") " pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:47.090862 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:47.090828    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn26m\" (UniqueName: \"kubernetes.io/projected/b467741e-e532-40ad-9261-db4f8b0cac24-kube-api-access-pn26m\") pod \"model-serving-api-75bd89fdfc-gbc6r\" (UID: \"b467741e-e532-40ad-9261-db4f8b0cac24\") " pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:47.240174 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:47.240084    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:47.547874 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:47.547842    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gbc6r"]
Oct 30 16:28:47.553796 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:28:47.553759    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb467741e_e532_40ad_9261_db4f8b0cac24.slice/crio-c122fe2938fff0e07c89ff7ba3ca93c67b7983d73a89096933871380f95c0851 WatchSource:0}: Error finding container c122fe2938fff0e07c89ff7ba3ca93c67b7983d73a89096933871380f95c0851: Status 404 returned error can't find the container with id c122fe2938fff0e07c89ff7ba3ca93c67b7983d73a89096933871380f95c0851
Oct 30 16:28:47.967097 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:47.967012    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r" event={"ID":"b467741e-e532-40ad-9261-db4f8b0cac24","Type":"ContainerStarted","Data":"c122fe2938fff0e07c89ff7ba3ca93c67b7983d73a89096933871380f95c0851"}
Oct 30 16:28:49.977782 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:49.977740    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r" event={"ID":"b467741e-e532-40ad-9261-db4f8b0cac24","Type":"ContainerStarted","Data":"495063a997832897d8a12b752d9ba0519cf87ec2b31ee4fd0a362271039736e1"}
Oct 30 16:28:49.978205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:49.977821    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:28:50.018026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:50.017979    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r" podStartSLOduration=2.419167878 podStartE2EDuration="4.017965144s" totalImagesPullingTime="1.598797266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:50.017402557 +0000 UTC m=+359.871280822" watchObservedRunningTime="2026-10-30 16:28:50.017965144 +0000 UTC m=+359.871843412"
Oct 30 16:28:50.309824 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:28:50.309791    4497 scope.go:122] "RemoveContainer" containerID="03e7ce2b77dd4a1a4de21b996c16a9141ee32c1e965b25f5d8f4d779d33679bd"
Oct 30 16:28:51.914939 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:28:51.914894    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:29:00.987084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:29:00.987040    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-gbc6r"
Oct 30 16:29:21.915384 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:29:21.915343    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:29:51.914605 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:29:51.914569    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:30:21.914502 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:30:21.914461    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:30:51.916527 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:30:51.916491    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:31:09.049239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.049142    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"]
Oct 30 16:31:09.052557 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.052538    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.067035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.067011    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 16:31:09.076335 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.076313    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 16:31:09.076335 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.076327    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b826n\""
Oct 30 16:31:09.076459 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.076327    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 16:31:09.078858 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.078841    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"]
Oct 30 16:31:09.181178 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.181143    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97f4feda-8781-4d4f-9dd5-77f0a5384990-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.181369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.181209    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7vf\" (UniqueName: \"kubernetes.io/projected/97f4feda-8781-4d4f-9dd5-77f0a5384990-kube-api-access-vj7vf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.181369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.181241    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.181439 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.181373    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.181439 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.181415    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.181520 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.181509    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.282693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.282658    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.282868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.282716    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97f4feda-8781-4d4f-9dd5-77f0a5384990-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.282868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.282737    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7vf\" (UniqueName: \"kubernetes.io/projected/97f4feda-8781-4d4f-9dd5-77f0a5384990-kube-api-access-vj7vf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.282868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.282756    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.282868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.282781    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.282868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.282798    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.283197 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.283165    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.283318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.283238    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.283318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.283297    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.285056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.285033    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97f4feda-8781-4d4f-9dd5-77f0a5384990-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.285142 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.285126    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97f4feda-8781-4d4f-9dd5-77f0a5384990-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.309923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.309859    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7vf\" (UniqueName: \"kubernetes.io/projected/97f4feda-8781-4d4f-9dd5-77f0a5384990-kube-api-access-vj7vf\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb\" (UID: \"97f4feda-8781-4d4f-9dd5-77f0a5384990\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.364850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.364814    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:09.669931 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:09.669849    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"]
Oct 30 16:31:09.673889 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:31:09.673865    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f4feda_8781_4d4f_9dd5_77f0a5384990.slice/crio-5b4dd573b7011a233481f7bc5d2f21c0841c7b72cc38bb4263cbcf3b561a6fdd WatchSource:0}: Error finding container 5b4dd573b7011a233481f7bc5d2f21c0841c7b72cc38bb4263cbcf3b561a6fdd: Status 404 returned error can't find the container with id 5b4dd573b7011a233481f7bc5d2f21c0841c7b72cc38bb4263cbcf3b561a6fdd
Oct 30 16:31:10.572158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:10.572119    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb" event={"ID":"97f4feda-8781-4d4f-9dd5-77f0a5384990","Type":"ContainerStarted","Data":"5b4dd573b7011a233481f7bc5d2f21c0841c7b72cc38bb4263cbcf3b561a6fdd"}
Oct 30 16:31:11.578431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:11.578397    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb" event={"ID":"97f4feda-8781-4d4f-9dd5-77f0a5384990","Type":"ContainerStarted","Data":"6e5c73cb88eb7bf99db0c1e74a75c29cfc61db0fbb4f41e55a3530d120bbe383"}
Oct 30 16:31:11.617548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:11.617493    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb" podStartSLOduration=2.456385869 podStartE2EDuration="3.617479497s" totalImagesPullingTime="1.161093628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:11.616731445 +0000 UTC m=+501.470609707" watchObservedRunningTime="2026-10-30 16:31:11.617479497 +0000 UTC m=+501.471357761"
Oct 30 16:31:12.299810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.299774    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"]
Oct 30 16:31:12.303268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.303248    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.317036 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.317008    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 16:31:12.330014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.329980    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"]
Oct 30 16:31:12.365853 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.365823    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:12.408311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.408278    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqtg\" (UniqueName: \"kubernetes.io/projected/55a8a020-fd10-4aa9-9b1b-12c100212c87-kube-api-access-6zqtg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.408489 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.408327    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.408489 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.408433    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.408562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.408533    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a8a020-fd10-4aa9-9b1b-12c100212c87-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.408596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.408571    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.408642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.408597    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.509549 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.509511    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.509743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.509564    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.509743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.509592    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a8a020-fd10-4aa9-9b1b-12c100212c87-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.509743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.509612    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.509743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.509629    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.509743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.509660    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqtg\" (UniqueName: \"kubernetes.io/projected/55a8a020-fd10-4aa9-9b1b-12c100212c87-kube-api-access-6zqtg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.510014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.509990    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.510014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.510007    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.510119 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.510075    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.512015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.511990    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55a8a020-fd10-4aa9-9b1b-12c100212c87-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.512255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.512236    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55a8a020-fd10-4aa9-9b1b-12c100212c87-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.531102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.531076    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqtg\" (UniqueName: \"kubernetes.io/projected/55a8a020-fd10-4aa9-9b1b-12c100212c87-kube-api-access-6zqtg\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-v4pm6\" (UID: \"55a8a020-fd10-4aa9-9b1b-12c100212c87\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.615618 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.615533    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:12.915948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:12.915870    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"]
Oct 30 16:31:12.921014 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:31:12.920980    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a8a020_fd10_4aa9_9b1b_12c100212c87.slice/crio-af2f3cf399cd541a26ddb6d5491cc1653603461b83511c55e7d951f52723b08b WatchSource:0}: Error finding container af2f3cf399cd541a26ddb6d5491cc1653603461b83511c55e7d951f52723b08b: Status 404 returned error can't find the container with id af2f3cf399cd541a26ddb6d5491cc1653603461b83511c55e7d951f52723b08b
Oct 30 16:31:13.589946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:13.589902    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6" event={"ID":"55a8a020-fd10-4aa9-9b1b-12c100212c87","Type":"ContainerStarted","Data":"e3f740095877c24e723bc19c59d0dd8e3ec9913b5a21162d6f6a55c9aa686336"}
Oct 30 16:31:13.590166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:13.589964    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6" event={"ID":"55a8a020-fd10-4aa9-9b1b-12c100212c87","Type":"ContainerStarted","Data":"af2f3cf399cd541a26ddb6d5491cc1653603461b83511c55e7d951f52723b08b"}
Oct 30 16:31:13.616660 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:13.616626    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:13.628818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:13.628764    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6" podStartSLOduration=1.4042920460000001 podStartE2EDuration="1.628750405s" totalImagesPullingTime="224.458359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:13.627144154 +0000 UTC m=+503.481022418" watchObservedRunningTime="2026-10-30 16:31:13.628750405 +0000 UTC m=+503.482628669"
Oct 30 16:31:18.274529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.274484    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"]
Oct 30 16:31:18.278681 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.278659    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.299845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.299821    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 16:31:18.311569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.311548    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"]
Oct 30 16:31:18.362717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.362677    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.362886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.362764    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.362886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.362822    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.362973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.362932    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.362973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.362966    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.363041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.362981    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jggkl\" (UniqueName: \"kubernetes.io/projected/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-kube-api-access-jggkl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.463608 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.463570    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.463608 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.463616    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.463870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.463636    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jggkl\" (UniqueName: \"kubernetes.io/projected/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-kube-api-access-jggkl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.463870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.463681    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.463870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.463710    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.463870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.463784    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.464134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.464069    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.464216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.464149    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.464216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.464158    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.466026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.466006    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.466173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.466154    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.484101 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.484079    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jggkl\" (UniqueName: \"kubernetes.io/projected/64cadc03-d87e-4a64-aa44-8e97d58c0dc5-kube-api-access-jggkl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq\" (UID: \"64cadc03-d87e-4a64-aa44-8e97d58c0dc5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.591503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.591418    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:18.891302 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:18.891217    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"]
Oct 30 16:31:18.895117 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:31:18.895091    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cadc03_d87e_4a64_aa44_8e97d58c0dc5.slice/crio-ad5c48d002e852bec9b7d052648438777fa6de043fe726b852a81e41a226e6ba WatchSource:0}: Error finding container ad5c48d002e852bec9b7d052648438777fa6de043fe726b852a81e41a226e6ba: Status 404 returned error can't find the container with id ad5c48d002e852bec9b7d052648438777fa6de043fe726b852a81e41a226e6ba
Oct 30 16:31:19.365957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:19.365921    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:19.378328 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:19.378303    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:19.616568 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:19.616533    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq" event={"ID":"64cadc03-d87e-4a64-aa44-8e97d58c0dc5","Type":"ContainerStarted","Data":"6d84f158a3d52ef5cd83893f0fdbf534a064298b2a376db4c643efa625b496c2"}
Oct 30 16:31:19.616773 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:19.616585    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq" event={"ID":"64cadc03-d87e-4a64-aa44-8e97d58c0dc5","Type":"ContainerStarted","Data":"ad5c48d002e852bec9b7d052648438777fa6de043fe726b852a81e41a226e6ba"}
Oct 30 16:31:19.627792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:19.627765    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb"
Oct 30 16:31:19.733445 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:19.733380    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq" podStartSLOduration=1.492056998 podStartE2EDuration="1.733362885s" totalImagesPullingTime="241.305887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:19.688801749 +0000 UTC m=+509.542680014" watchObservedRunningTime="2026-10-30 16:31:19.733362885 +0000 UTC m=+509.587241152"
Oct 30 16:31:20.591648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:20.591612    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:21.914605 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:31:21.914570    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:31:22.616363 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:22.616331    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:22.635094 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:22.635060    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:22.645981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:22.645958    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-v4pm6"
Oct 30 16:31:26.031072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.031038    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"]
Oct 30 16:31:26.038143 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.038117    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.066220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.066171    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 16:31:26.067381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.067360    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"]
Oct 30 16:31:26.235909 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.235875    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.235909 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.235905    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e27b12c-daa9-49eb-9f5a-0815ec044446-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.236123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.235929    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.236123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.236044    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.236123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.236078    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthmw\" (UniqueName: \"kubernetes.io/projected/5e27b12c-daa9-49eb-9f5a-0815ec044446-kube-api-access-wthmw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.236263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.236179    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.271447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.271412    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"]
Oct 30 16:31:26.275464 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.275444    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.293596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.293575    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 16:31:26.293725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.293695    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xtm6x\""
Oct 30 16:31:26.300442 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.300418    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"]
Oct 30 16:31:26.337002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.336977    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.337198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.337009    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.337198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.337027    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e27b12c-daa9-49eb-9f5a-0815ec044446-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.337198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.337047    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.337198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.337086    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.337198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.337107    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthmw\" (UniqueName: \"kubernetes.io/projected/5e27b12c-daa9-49eb-9f5a-0815ec044446-kube-api-access-wthmw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.337471 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.337413    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.337530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.337467    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.337530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.337482    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.339303 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.339281    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e27b12c-daa9-49eb-9f5a-0815ec044446-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.339555 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.339540    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e27b12c-daa9-49eb-9f5a-0815ec044446-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.359545 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.359522    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthmw\" (UniqueName: \"kubernetes.io/projected/5e27b12c-daa9-49eb-9f5a-0815ec044446-kube-api-access-wthmw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x\" (UID: \"5e27b12c-daa9-49eb-9f5a-0815ec044446\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.438433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438388    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.438609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438474    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.438609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438577    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p82j\" (UniqueName: \"kubernetes.io/projected/c92b23e0-1211-499e-a41d-ab0ae8387659-kube-api-access-9p82j\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.438686 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438651    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.438722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438690    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.438777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438758    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.438812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438795    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.438847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438811    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92b23e0-1211-499e-a41d-ab0ae8387659-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.438885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.438873    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.539729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539628    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.539729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539684    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p82j\" (UniqueName: \"kubernetes.io/projected/c92b23e0-1211-499e-a41d-ab0ae8387659-kube-api-access-9p82j\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.539729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539712    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.539729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539738    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539757    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539786    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539809    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92b23e0-1211-499e-a41d-ab0ae8387659-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539856    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.539904    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540337 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.540106    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540337 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.540152    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540337 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.540277    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540337 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.540325    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.540546 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.540525    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92b23e0-1211-499e-a41d-ab0ae8387659-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.542241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.542211    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.542435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.542412    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.568058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.568027    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92b23e0-1211-499e-a41d-ab0ae8387659-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.569113 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.569090    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p82j\" (UniqueName: \"kubernetes.io/projected/c92b23e0-1211-499e-a41d-ab0ae8387659-kube-api-access-9p82j\") pod \"maas-default-gateway-openshift-default-7854949c9-jzt7r\" (UID: \"c92b23e0-1211-499e-a41d-ab0ae8387659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.588022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.587999    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:26.654390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.654364    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:26.909910 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.909831    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"]
Oct 30 16:31:26.926105 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:31:26.926081    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92b23e0_1211_499e_a41d_ab0ae8387659.slice/crio-b77fb9bcf5ee3e20054ef0d30a8621ecf35d9536d47e54f727bd1e31dc0f6edb WatchSource:0}: Error finding container b77fb9bcf5ee3e20054ef0d30a8621ecf35d9536d47e54f727bd1e31dc0f6edb: Status 404 returned error can't find the container with id b77fb9bcf5ee3e20054ef0d30a8621ecf35d9536d47e54f727bd1e31dc0f6edb
Oct 30 16:31:26.954229 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:26.954200    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"]
Oct 30 16:31:26.958840 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:31:26.958817    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e27b12c_daa9_49eb_9f5a_0815ec044446.slice/crio-e5305319f451376a914805660b34f2f1538e37e8b295cb2da324bb68b169a89c WatchSource:0}: Error finding container e5305319f451376a914805660b34f2f1538e37e8b295cb2da324bb68b169a89c: Status 404 returned error can't find the container with id e5305319f451376a914805660b34f2f1538e37e8b295cb2da324bb68b169a89c
Oct 30 16:31:27.655677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:27.655641    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" event={"ID":"c92b23e0-1211-499e-a41d-ab0ae8387659","Type":"ContainerStarted","Data":"b77fb9bcf5ee3e20054ef0d30a8621ecf35d9536d47e54f727bd1e31dc0f6edb"}
Oct 30 16:31:27.658030 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:27.657657    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x" event={"ID":"5e27b12c-daa9-49eb-9f5a-0815ec044446","Type":"ContainerStarted","Data":"0248bfe8dacbdaa8fe3e4b2b34a61eb468f43bd481b7759f14838cd4f56f61d8"}
Oct 30 16:31:27.658030 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:27.657709    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x" event={"ID":"5e27b12c-daa9-49eb-9f5a-0815ec044446","Type":"ContainerStarted","Data":"e5305319f451376a914805660b34f2f1538e37e8b295cb2da324bb68b169a89c"}
Oct 30 16:31:27.698776 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:27.698726    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x" podStartSLOduration=1.496241631 podStartE2EDuration="1.698709201s" totalImagesPullingTime="202.46757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:27.697055892 +0000 UTC m=+517.550934158" watchObservedRunningTime="2026-10-30 16:31:27.698709201 +0000 UTC m=+517.552587512"
Oct 30 16:31:28.591696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:28.591652    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:28.605717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:28.605685    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:28.655283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:28.655245    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:28.676524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:28.676496    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq"
Oct 30 16:31:29.417541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:29.417508    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:31:29.417655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:29.417595    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:31:29.417655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:29.417626    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:31:29.670077 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:29.669988    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" event={"ID":"c92b23e0-1211-499e-a41d-ab0ae8387659","Type":"ContainerStarted","Data":"88ffb12a87c6d8d67978f5e67c86d0685de6987cf2e673becf25812421b6709d"}
Oct 30 16:31:29.731378 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:29.731321    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podStartSLOduration=1.242806253 podStartE2EDuration="3.731306738s" totalImagesPullingTime="2.488500485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:29.720959797 +0000 UTC m=+519.574838061" watchObservedRunningTime="2026-10-30 16:31:29.731306738 +0000 UTC m=+519.585185073"
Oct 30 16:31:30.588702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:30.588662    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:30.592874 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:30.592838    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jzt7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:31:30.593148 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:30.593116    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podUID="c92b23e0-1211-499e-a41d-ab0ae8387659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:31:31.596240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:31.591626    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jzt7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:31:31.596240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:31.591692    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podUID="c92b23e0-1211-499e-a41d-ab0ae8387659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:31:32.591212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:32.588634    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jzt7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:31:32.591212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:32.588700    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podUID="c92b23e0-1211-499e-a41d-ab0ae8387659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:31:33.591671 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:33.591630    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jzt7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:31:33.592322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:33.592290    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podUID="c92b23e0-1211-499e-a41d-ab0ae8387659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:31:34.596147 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:34.595632    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jzt7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:31:34.596147 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:34.595697    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podUID="c92b23e0-1211-499e-a41d-ab0ae8387659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:31:35.589242 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:35.589164    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jzt7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:31:35.589419 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:35.589268    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podUID="c92b23e0-1211-499e-a41d-ab0ae8387659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:31:36.588603 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:36.588551    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jzt7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 16:31:36.589130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:36.588612    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podUID="c92b23e0-1211-499e-a41d-ab0ae8387659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 16:31:36.655143 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:36.655106    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:36.668514 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:36.668479    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:36.731974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:36.731945    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x"
Oct 30 16:31:37.661822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:37.661791    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:37.662427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:37.662410    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:37.663120 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:37.663093    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r"
Oct 30 16:31:51.916519 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:31:51.916482    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:31:56.363731 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.363687    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"]
Oct 30 16:31:56.367940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.367910    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.395784 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.395748    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"]
Oct 30 16:31:56.436048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436014    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.436266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436054    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbppm\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-kube-api-access-mbppm\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.436266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436085    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.436266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436250    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.436401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436294    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.436401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436323    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.436401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436356    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.436555 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436465    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.436609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.436575    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537020    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537058    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbppm\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-kube-api-access-mbppm\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537086    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537224    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537257    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537287    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537332    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537377    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537415    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537544    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537610    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537649    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.537801 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537741    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.538020 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.537997    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.539476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.539458    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.539742 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.539727    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.558417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.558386    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.558568 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.558549    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbppm\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-kube-api-access-mbppm\") pod \"maas-default-gateway-openshift-default-7854949c9-pkh5z\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:56.691075 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:56.690990    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:57.012518 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:57.012485    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"]
Oct 30 16:31:57.030682 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:31:57.030646    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bdffc5_6351_41e4_86dd_f7e438c4be4e.slice/crio-5003212922f2dfe2d25ae2cd064161a4ec82636b06e893c88306a5c0fcae7bb8 WatchSource:0}: Error finding container 5003212922f2dfe2d25ae2cd064161a4ec82636b06e893c88306a5c0fcae7bb8: Status 404 returned error can't find the container with id 5003212922f2dfe2d25ae2cd064161a4ec82636b06e893c88306a5c0fcae7bb8
Oct 30 16:31:57.033056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:57.033023    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:31:57.033118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:57.033092    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:31:57.033163 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:57.033120    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:31:57.816394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:57.816359    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" event={"ID":"b8bdffc5-6351-41e4-86dd-f7e438c4be4e","Type":"ContainerStarted","Data":"5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5"}
Oct 30 16:31:57.816786 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:57.816414    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" event={"ID":"b8bdffc5-6351-41e4-86dd-f7e438c4be4e","Type":"ContainerStarted","Data":"5003212922f2dfe2d25ae2cd064161a4ec82636b06e893c88306a5c0fcae7bb8"}
Oct 30 16:31:57.889212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:57.886956    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podStartSLOduration=1.88693537 podStartE2EDuration="1.88693537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:57.866765922 +0000 UTC m=+547.720644186" watchObservedRunningTime="2026-10-30 16:31:57.88693537 +0000 UTC m=+547.740813635"
Oct 30 16:31:58.696492 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:58.696457    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:31:58.698031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:58.698003    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-pkh5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 16:31:58.698160 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:58.698052    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 16:31:59.695212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:59.694324    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-pkh5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 16:31:59.695212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:31:59.694484    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 16:32:00.696635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:00.696594    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-pkh5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 16:32:00.697124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:00.696662    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 16:32:01.692627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:01.692586    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-pkh5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 16:32:01.692922 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:01.692891    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 16:32:02.693212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:02.692732    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-pkh5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 16:32:02.693212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:02.692806    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 16:32:03.691708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:03.691669    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-pkh5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 16:32:03.691902 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:03.691775    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 16:32:05.246764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:05.246733    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:32:05.247265 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:05.247023    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:32:05.247839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:05.247818    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:32:15.185882 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.185850    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"]
Oct 30 16:32:15.189503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.189481    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.203970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.203948    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cff59996-openshift-default-dockercfg-mwzbn\""
Oct 30 16:32:15.220880 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.220855    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"]
Oct 30 16:32:15.234612 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234586    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-data\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.234780 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234620    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-istio-token\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.234780 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234700    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-envoy\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.234780 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234725    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-credential-socket\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.234780 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234744    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-socket\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.235008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234814    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-certs\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.235008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234865    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdxv\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-kube-api-access-jwdxv\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.235008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234907    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e20db3-a768-4ba9-8588-24043f52043f-istiod-ca-cert\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.235008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.234976    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e20db3-a768-4ba9-8588-24043f52043f-istio-podinfo\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.306896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.306861    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"]
Oct 30 16:32:15.310529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.310504    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.325743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.325710    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f756b7bf-openshift-default-dockercfg-pjbx5\""
Oct 30 16:32:15.335968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.335936    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-envoy\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.336137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.335978    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-credential-socket\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.336137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336001    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-data\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.336137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336025    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-token\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.336137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336053    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-socket\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.336392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336154    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-certs\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.336392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336373    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-envoy\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.336506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336413    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdxv\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-kube-api-access-jwdxv\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.336506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336449    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e20db3-a768-4ba9-8588-24043f52043f-istiod-ca-cert\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.336506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336488    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-credential-socket\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.336839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336514    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-certs\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.336839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336554    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-socket\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.336839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336601    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e20db3-a768-4ba9-8588-24043f52043f-istio-podinfo\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.336839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336628    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-podinfo\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.336839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336680    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-certs\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.336839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336686    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483fdd8-8671-40ed-b785-e54e2a0548d1-istiod-ca-cert\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.336839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.336767    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-socket\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.337321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.337295    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"]
Oct 30 16:32:15.337678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.337415    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-data\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.337678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.337416    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e20db3-a768-4ba9-8588-24043f52043f-istiod-ca-cert\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.337678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.337593    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-istio-token\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.337678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.337660    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9775\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-kube-api-access-f9775\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.337678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.337673    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-data\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.338722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.338660    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-credential-socket\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.340454 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.340424    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-envoy\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.340609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.340588    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e20db3-a768-4ba9-8588-24043f52043f-istio-podinfo\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.362544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.362507    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-istio-token\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.365202 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.365163    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdxv\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-kube-api-access-jwdxv\") pod \"e2e-ait-cff59996-openshift-default-55f88d9fff-572z4\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") " pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.439152 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439056    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9775\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-kube-api-access-f9775\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439152 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439131    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-data\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439154    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-token\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439238    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-envoy\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439280    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-credential-socket\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439416    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-certs\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439467    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-socket\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439513    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-podinfo\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439568    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483fdd8-8671-40ed-b785-e54e2a0548d1-istiod-ca-cert\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439590    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-data\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439649    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-credential-socket\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439757    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-certs\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.439917 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.439822    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-socket\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.440139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.440117    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483fdd8-8671-40ed-b785-e54e2a0548d1-istiod-ca-cert\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.441666 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.441641    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-envoy\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.441860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.441842    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-podinfo\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.460639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.460608    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-token\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.460768 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.460677    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9775\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-kube-api-access-f9775\") pod \"e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") " pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.503129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.503095    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:15.626370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.626322    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:15.829577 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.829544    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"]
Oct 30 16:32:15.848135 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:15.848097    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e20db3_a768_4ba9_8588_24043f52043f.slice/crio-51ef29ca8eda0c7f799d3e64d1522c55c3f9445f420a7cf106ac95cd7720d574 WatchSource:0}: Error finding container 51ef29ca8eda0c7f799d3e64d1522c55c3f9445f420a7cf106ac95cd7720d574: Status 404 returned error can't find the container with id 51ef29ca8eda0c7f799d3e64d1522c55c3f9445f420a7cf106ac95cd7720d574
Oct 30 16:32:15.850342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.850308    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:15.850427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.850376    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:15.850427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.850406    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:15.901075 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.901038    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" event={"ID":"63e20db3-a768-4ba9-8588-24043f52043f","Type":"ContainerStarted","Data":"51ef29ca8eda0c7f799d3e64d1522c55c3f9445f420a7cf106ac95cd7720d574"}
Oct 30 16:32:15.941304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.941179    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"]
Oct 30 16:32:15.945710 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:15.945670    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5483fdd8_8671_40ed_b785_e54e2a0548d1.slice/crio-821eee6c387fc5f861bd5dd3d1b02a208193d2c1180e64b262df0b7889da5911 WatchSource:0}: Error finding container 821eee6c387fc5f861bd5dd3d1b02a208193d2c1180e64b262df0b7889da5911: Status 404 returned error can't find the container with id 821eee6c387fc5f861bd5dd3d1b02a208193d2c1180e64b262df0b7889da5911
Oct 30 16:32:15.948003 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.947960    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:15.948085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.948053    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:15.948124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:15.948095    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:16.915223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:16.908054    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" event={"ID":"63e20db3-a768-4ba9-8588-24043f52043f","Type":"ContainerStarted","Data":"f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5"}
Oct 30 16:32:16.915223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:16.909971    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" event={"ID":"5483fdd8-8671-40ed-b785-e54e2a0548d1","Type":"ContainerStarted","Data":"c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5"}
Oct 30 16:32:16.915223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:16.910021    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" event={"ID":"5483fdd8-8671-40ed-b785-e54e2a0548d1","Type":"ContainerStarted","Data":"821eee6c387fc5f861bd5dd3d1b02a208193d2c1180e64b262df0b7889da5911"}
Oct 30 16:32:16.956873 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:16.956828    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" podStartSLOduration=1.9568163699999999 podStartE2EDuration="1.95681637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:16.952167544 +0000 UTC m=+566.806045809" watchObservedRunningTime="2026-10-30 16:32:16.95681637 +0000 UTC m=+566.810694635"
Oct 30 16:32:16.994069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:16.994009    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" podStartSLOduration=1.993993821 podStartE2EDuration="1.993993821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:16.992792478 +0000 UTC m=+566.846670736" watchObservedRunningTime="2026-10-30 16:32:16.993993821 +0000 UTC m=+566.847872086"
Oct 30 16:32:17.503396 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:17.503362    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:17.508430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:17.508403    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:17.626774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:17.626737    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:17.631637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:17.631607    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:17.914694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:17.914655    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:17.914891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:17.914711    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:17.915626 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:17.915601    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:17.916008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:17.915674    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:21.179196 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.179160    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"]
Oct 30 16:32:21.183321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.183296    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:21.196972 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.196941    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f756b7bf\""
Oct 30 16:32:21.206651 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.206624    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f756b7bf-dockercfg-9m2rr\""
Oct 30 16:32:21.209579 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.209516    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"]
Oct 30 16:32:21.264216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.264142    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"]
Oct 30 16:32:21.268166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.268139    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:21.289065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.289038    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"]
Oct 30 16:32:21.301961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.301901    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc72be2b-3fd6-4e02-a30e-4154656cb651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl\" (UID: \"bc72be2b-3fd6-4e02-a30e-4154656cb651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:21.302162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.302028    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzrm\" (UniqueName: \"kubernetes.io/projected/bc72be2b-3fd6-4e02-a30e-4154656cb651-kube-api-access-2tzrm\") pod \"payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl\" (UID: \"bc72be2b-3fd6-4e02-a30e-4154656cb651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:21.402957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.402916    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5xq\" (UniqueName: \"kubernetes.io/projected/244468e3-f44c-4055-af6f-3f6474547ea0-kube-api-access-zs5xq\") pod \"payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm\" (UID: \"244468e3-f44c-4055-af6f-3f6474547ea0\") " pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:21.402957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.402965    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzrm\" (UniqueName: \"kubernetes.io/projected/bc72be2b-3fd6-4e02-a30e-4154656cb651-kube-api-access-2tzrm\") pod \"payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl\" (UID: \"bc72be2b-3fd6-4e02-a30e-4154656cb651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:21.403173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.403148    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc72be2b-3fd6-4e02-a30e-4154656cb651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl\" (UID: \"bc72be2b-3fd6-4e02-a30e-4154656cb651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:21.403248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.403213    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244468e3-f44c-4055-af6f-3f6474547ea0-plugins-config-volume\") pod \"payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm\" (UID: \"244468e3-f44c-4055-af6f-3f6474547ea0\") " pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:21.403814 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.403797    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc72be2b-3fd6-4e02-a30e-4154656cb651-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl\" (UID: \"bc72be2b-3fd6-4e02-a30e-4154656cb651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:21.423235 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.423177    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzrm\" (UniqueName: \"kubernetes.io/projected/bc72be2b-3fd6-4e02-a30e-4154656cb651-kube-api-access-2tzrm\") pod \"payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl\" (UID: \"bc72be2b-3fd6-4e02-a30e-4154656cb651\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:21.496297 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.496205    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:21.503706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.503681    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244468e3-f44c-4055-af6f-3f6474547ea0-plugins-config-volume\") pod \"payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm\" (UID: \"244468e3-f44c-4055-af6f-3f6474547ea0\") " pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:21.503819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.503731    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5xq\" (UniqueName: \"kubernetes.io/projected/244468e3-f44c-4055-af6f-3f6474547ea0-kube-api-access-zs5xq\") pod \"payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm\" (UID: \"244468e3-f44c-4055-af6f-3f6474547ea0\") " pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:21.504565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.504544    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244468e3-f44c-4055-af6f-3f6474547ea0-plugins-config-volume\") pod \"payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm\" (UID: \"244468e3-f44c-4055-af6f-3f6474547ea0\") " pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:21.524890 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.524862    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5xq\" (UniqueName: \"kubernetes.io/projected/244468e3-f44c-4055-af6f-3f6474547ea0-kube-api-access-zs5xq\") pod \"payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm\" (UID: \"244468e3-f44c-4055-af6f-3f6474547ea0\") " pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:21.585800 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.585765    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:21.799659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.799627    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"]
Oct 30 16:32:21.803994 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:21.803965    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc72be2b_3fd6_4e02_a30e_4154656cb651.slice/crio-19b4ce15de2d3d05be409b28d2798a3952c72fe66fe97f27a1fd741cac7e294b WatchSource:0}: Error finding container 19b4ce15de2d3d05be409b28d2798a3952c72fe66fe97f27a1fd741cac7e294b: Status 404 returned error can't find the container with id 19b4ce15de2d3d05be409b28d2798a3952c72fe66fe97f27a1fd741cac7e294b
Oct 30 16:32:21.889406 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.889372    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"]
Oct 30 16:32:21.895178 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:21.895134    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244468e3_f44c_4055_af6f_3f6474547ea0.slice/crio-f938343baf6a4eca5c29a9bec69d5679cef605399ea041749a4072e35cfcb7c7 WatchSource:0}: Error finding container f938343baf6a4eca5c29a9bec69d5679cef605399ea041749a4072e35cfcb7c7: Status 404 returned error can't find the container with id f938343baf6a4eca5c29a9bec69d5679cef605399ea041749a4072e35cfcb7c7
Oct 30 16:32:21.918321 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:32:21.918286    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:32:21.934640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.934608    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm" event={"ID":"244468e3-f44c-4055-af6f-3f6474547ea0","Type":"ContainerStarted","Data":"f938343baf6a4eca5c29a9bec69d5679cef605399ea041749a4072e35cfcb7c7"}
Oct 30 16:32:21.935760 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:21.935735    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl" event={"ID":"bc72be2b-3fd6-4e02-a30e-4154656cb651","Type":"ContainerStarted","Data":"19b4ce15de2d3d05be409b28d2798a3952c72fe66fe97f27a1fd741cac7e294b"}
Oct 30 16:32:22.111696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.111611    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"]
Oct 30 16:32:22.116802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.116777    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:22.131650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.131626    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cff59996\""
Oct 30 16:32:22.131650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.131639    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cff59996-dockercfg-xpmrm\""
Oct 30 16:32:22.148889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.148854    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"]
Oct 30 16:32:22.217026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.216798    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqxm\" (UniqueName: \"kubernetes.io/projected/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-kube-api-access-ttqxm\") pod \"payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9\" (UID: \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:22.222243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.220685    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"]
Oct 30 16:32:22.223294 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.223267    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9\" (UID: \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:22.227693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.227663    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:22.244556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.244530    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"]
Oct 30 16:32:22.324549 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.324517    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e980934-b9c1-4cf6-b93f-b4397dbe8753-plugins-config-volume\") pod \"payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn\" (UID: \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\") " pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:22.324727 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.324657    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnb9h\" (UniqueName: \"kubernetes.io/projected/5e980934-b9c1-4cf6-b93f-b4397dbe8753-kube-api-access-gnb9h\") pod \"payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn\" (UID: \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\") " pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:22.324727 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.324714    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqxm\" (UniqueName: \"kubernetes.io/projected/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-kube-api-access-ttqxm\") pod \"payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9\" (UID: \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:22.324804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.324763    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9\" (UID: \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:22.325450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.325431    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9\" (UID: \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:22.346307 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.346280    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqxm\" (UniqueName: \"kubernetes.io/projected/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-kube-api-access-ttqxm\") pod \"payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9\" (UID: \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:22.438861 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.438775    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnb9h\" (UniqueName: \"kubernetes.io/projected/5e980934-b9c1-4cf6-b93f-b4397dbe8753-kube-api-access-gnb9h\") pod \"payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn\" (UID: \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\") " pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:22.439025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.438892    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e980934-b9c1-4cf6-b93f-b4397dbe8753-plugins-config-volume\") pod \"payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn\" (UID: \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\") " pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:22.439791 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.439764    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e980934-b9c1-4cf6-b93f-b4397dbe8753-plugins-config-volume\") pod \"payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn\" (UID: \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\") " pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:22.442023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.442000    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:22.469981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.469949    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnb9h\" (UniqueName: \"kubernetes.io/projected/5e980934-b9c1-4cf6-b93f-b4397dbe8753-kube-api-access-gnb9h\") pod \"payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn\" (UID: \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\") " pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:22.542709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.541389    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:22.791314 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.791278    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"]
Oct 30 16:32:22.798746 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:22.797296    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548f988e_1e4a_4017_a5b3_0377cb8d7c7b.slice/crio-58cdd760354cf1c10a119a878a14f3c8c73794db807459b209b6091118491f50 WatchSource:0}: Error finding container 58cdd760354cf1c10a119a878a14f3c8c73794db807459b209b6091118491f50: Status 404 returned error can't find the container with id 58cdd760354cf1c10a119a878a14f3c8c73794db807459b209b6091118491f50
Oct 30 16:32:22.881652 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.880679    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"]
Oct 30 16:32:22.885828 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:22.885783    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e980934_b9c1_4cf6_b93f_b4397dbe8753.slice/crio-1fc75a8b3b494ad32fe79779d3f0fd781e798c3643a88a0baeaa5edf57b107e0 WatchSource:0}: Error finding container 1fc75a8b3b494ad32fe79779d3f0fd781e798c3643a88a0baeaa5edf57b107e0: Status 404 returned error can't find the container with id 1fc75a8b3b494ad32fe79779d3f0fd781e798c3643a88a0baeaa5edf57b107e0
Oct 30 16:32:22.944373 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.944334    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9" event={"ID":"548f988e-1e4a-4017-a5b3-0377cb8d7c7b","Type":"ContainerStarted","Data":"58cdd760354cf1c10a119a878a14f3c8c73794db807459b209b6091118491f50"}
Oct 30 16:32:22.945897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:22.945867    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn" event={"ID":"5e980934-b9c1-4cf6-b93f-b4397dbe8753","Type":"ContainerStarted","Data":"1fc75a8b3b494ad32fe79779d3f0fd781e798c3643a88a0baeaa5edf57b107e0"}
Oct 30 16:32:24.641004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.640966    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"]
Oct 30 16:32:24.649587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.648693    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:24.669291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.669214    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a2d47c16\""
Oct 30 16:32:24.682087 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.682051    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"]
Oct 30 16:32:24.682283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.682259    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a2d47c16-dockercfg-st8rg\""
Oct 30 16:32:24.726211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.725851    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"]
Oct 30 16:32:24.736521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.736489    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:24.756142 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.756060    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"]
Oct 30 16:32:24.784001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.783967    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7hh\" (UniqueName: \"kubernetes.io/projected/34849fd3-9153-4a70-85d6-a0bfd562625f-kube-api-access-wr7hh\") pod \"payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r\" (UID: \"34849fd3-9153-4a70-85d6-a0bfd562625f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:24.784228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.784027    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34849fd3-9153-4a70-85d6-a0bfd562625f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r\" (UID: \"34849fd3-9153-4a70-85d6-a0bfd562625f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:24.885169 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.885130    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg24g\" (UniqueName: \"kubernetes.io/projected/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-kube-api-access-dg24g\") pod \"payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6\" (UID: \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\") " pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:24.885375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.885241    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7hh\" (UniqueName: \"kubernetes.io/projected/34849fd3-9153-4a70-85d6-a0bfd562625f-kube-api-access-wr7hh\") pod \"payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r\" (UID: \"34849fd3-9153-4a70-85d6-a0bfd562625f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:24.885375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.885288    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34849fd3-9153-4a70-85d6-a0bfd562625f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r\" (UID: \"34849fd3-9153-4a70-85d6-a0bfd562625f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:24.885375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.885307    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6\" (UID: \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\") " pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:24.886269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.886222    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34849fd3-9153-4a70-85d6-a0bfd562625f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r\" (UID: \"34849fd3-9153-4a70-85d6-a0bfd562625f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:24.912262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.912129    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7hh\" (UniqueName: \"kubernetes.io/projected/34849fd3-9153-4a70-85d6-a0bfd562625f-kube-api-access-wr7hh\") pod \"payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r\" (UID: \"34849fd3-9153-4a70-85d6-a0bfd562625f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:24.982109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.982070    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:24.986911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.986883    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg24g\" (UniqueName: \"kubernetes.io/projected/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-kube-api-access-dg24g\") pod \"payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6\" (UID: \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\") " pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:24.987050 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.987005    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6\" (UID: \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\") " pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:24.987849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:24.987801    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6\" (UID: \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\") " pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:25.008621 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.008595    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg24g\" (UniqueName: \"kubernetes.io/projected/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-kube-api-access-dg24g\") pod \"payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6\" (UID: \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\") " pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:25.057921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.057712    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:25.325437 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.325397    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"]
Oct 30 16:32:25.419103 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.419022    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"]
Oct 30 16:32:25.429339 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:25.429304    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40a2f74_c18c_4c9a_b726_2a20da7aa4ac.slice/crio-178b56b2fa575710e02546a6ff492f31bf8f7eddc55d446cec6924f2c20cab3c WatchSource:0}: Error finding container 178b56b2fa575710e02546a6ff492f31bf8f7eddc55d446cec6924f2c20cab3c: Status 404 returned error can't find the container with id 178b56b2fa575710e02546a6ff492f31bf8f7eddc55d446cec6924f2c20cab3c
Oct 30 16:32:25.967397 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.967362    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn" event={"ID":"5e980934-b9c1-4cf6-b93f-b4397dbe8753","Type":"ContainerStarted","Data":"fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d"}
Oct 30 16:32:25.967850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.967497    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:25.968946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.968924    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm" event={"ID":"244468e3-f44c-4055-af6f-3f6474547ea0","Type":"ContainerStarted","Data":"e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643"}
Oct 30 16:32:25.969065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.968970    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:25.970461 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.970439    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6" event={"ID":"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac","Type":"ContainerStarted","Data":"22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924"}
Oct 30 16:32:25.970555 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.970475    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6" event={"ID":"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac","Type":"ContainerStarted","Data":"178b56b2fa575710e02546a6ff492f31bf8f7eddc55d446cec6924f2c20cab3c"}
Oct 30 16:32:25.970555 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.970543    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:25.971892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.971871    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r" event={"ID":"34849fd3-9153-4a70-85d6-a0bfd562625f","Type":"ContainerStarted","Data":"6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683"}
Oct 30 16:32:25.971996 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.971907    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r" event={"ID":"34849fd3-9153-4a70-85d6-a0bfd562625f","Type":"ContainerStarted","Data":"e93ecb6c9c8adab21f5a86ae8a7703cd7a44eab7544a0238cf3711e262401a1a"}
Oct 30 16:32:25.971996 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.971934    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:25.973276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.973258    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl" event={"ID":"bc72be2b-3fd6-4e02-a30e-4154656cb651","Type":"ContainerStarted","Data":"83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6"}
Oct 30 16:32:25.973358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.973334    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:25.974618 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.974601    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9" event={"ID":"548f988e-1e4a-4017-a5b3-0377cb8d7c7b","Type":"ContainerStarted","Data":"7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17"}
Oct 30 16:32:25.974726 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:25.974715    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:26.005756 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:26.005692    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn" podStartSLOduration=1.333555856 podStartE2EDuration="4.005674173s" totalImagesPullingTime="2.672118317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:26.00526377 +0000 UTC m=+575.859142036" watchObservedRunningTime="2026-10-30 16:32:26.005674173 +0000 UTC m=+575.859552439"
Oct 30 16:32:26.040813 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:26.040593    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6" podStartSLOduration=2.040575795 podStartE2EDuration="2.040575795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:26.040076499 +0000 UTC m=+575.893954766" watchObservedRunningTime="2026-10-30 16:32:26.040575795 +0000 UTC m=+575.894454059"
Oct 30 16:32:26.076458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:26.076404    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm" podStartSLOduration=1.938607829 podStartE2EDuration="5.076389922s" totalImagesPullingTime="3.137782093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:26.074619667 +0000 UTC m=+575.928497932" watchObservedRunningTime="2026-10-30 16:32:26.076389922 +0000 UTC m=+575.930268187"
Oct 30 16:32:26.109294 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:26.109236    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r" podStartSLOduration=2.109223515 podStartE2EDuration="2.109223515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:26.108963167 +0000 UTC m=+575.962841432" watchObservedRunningTime="2026-10-30 16:32:26.109223515 +0000 UTC m=+575.963101780"
Oct 30 16:32:26.146197 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:26.146144    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl" podStartSLOduration=1.910223635 podStartE2EDuration="5.146129788s" totalImagesPullingTime="3.235906153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:26.144090352 +0000 UTC m=+575.997968609" watchObservedRunningTime="2026-10-30 16:32:26.146129788 +0000 UTC m=+576.000008053"
Oct 30 16:32:26.180679 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:26.180619    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9" podStartSLOduration=1.390401489 podStartE2EDuration="4.180602485s" totalImagesPullingTime="2.790200996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:26.178830827 +0000 UTC m=+576.032709106" watchObservedRunningTime="2026-10-30 16:32:26.180602485 +0000 UTC m=+576.034480749"
Oct 30 16:32:36.999472 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:36.999429    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:37.000207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.000168    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:37.000409 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.000396    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:32:37.001144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.001128    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:32:37.001542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.001523    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:37.002432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.002408    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:37.130481 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.130434    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"]
Oct 30 16:32:37.135954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.135921    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.154272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.154236    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 16:32:37.154272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.154250    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 16:32:37.154701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.154681    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 16:32:37.154791 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.154736    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k46zs\""
Oct 30 16:32:37.154791 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.154752    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a3cc1e2b-serving-cert\""
Oct 30 16:32:37.166458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.166380    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"]
Oct 30 16:32:37.224694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.224659    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5bjz\" (UniqueName: \"kubernetes.io/projected/d8a7e1af-b485-4226-86c3-fa1bc739c14b-kube-api-access-z5bjz\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.224892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.224734    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.224969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.224903    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-metrics-certs\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.224969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.224936    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a7e1af-b485-4226-86c3-fa1bc739c14b-openshift-service-ca\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.326309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.326276    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5bjz\" (UniqueName: \"kubernetes.io/projected/d8a7e1af-b485-4226-86c3-fa1bc739c14b-kube-api-access-z5bjz\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.326506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.326349    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.326506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.326397    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-metrics-certs\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.326506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.326419    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a7e1af-b485-4226-86c3-fa1bc739c14b-openshift-service-ca\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.326703 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:32:37.326512    4497 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-serving-cert: secret "maas-api-e2e-derive-a3cc1e2b-serving-cert" not found
Oct 30 16:32:37.326703 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:32:37.326598    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls podName:d8a7e1af-b485-4226-86c3-fa1bc739c14b nodeName:}" failed. No retries permitted until 2026-10-30 16:32:37.826576286 +0000 UTC m=+587.680454528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls") pod "maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn" (UID: "d8a7e1af-b485-4226-86c3-fa1bc739c14b") : secret "maas-api-e2e-derive-a3cc1e2b-serving-cert" not found
Oct 30 16:32:37.327047 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.327024    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a7e1af-b485-4226-86c3-fa1bc739c14b-openshift-service-ca\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.330790 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.330765    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-metrics-certs\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.347700 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.347674    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5bjz\" (UniqueName: \"kubernetes.io/projected/d8a7e1af-b485-4226-86c3-fa1bc739c14b-kube-api-access-z5bjz\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.831952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.831912    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:37.834434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:37.834400    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls\") pod \"maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:38.042553 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:38.042521    4497 generic.go:356] "Generic (PLEG): container finished" podID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerID="c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5" exitCode=0
Oct 30 16:32:38.042962 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:38.042596    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" event={"ID":"5483fdd8-8671-40ed-b785-e54e2a0548d1","Type":"ContainerDied","Data":"c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5"}
Oct 30 16:32:38.043003 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:38.042976    4497 scope.go:122] "RemoveContainer" containerID="c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5"
Oct 30 16:32:38.050571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:38.044817    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:38.050571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:38.044989    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:38.050571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:38.045089    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:38.051076 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:38.051037    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:38.405336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:38.405302    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"]
Oct 30 16:32:38.406581 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:38.406550    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a7e1af_b485_4226_86c3_fa1bc739c14b.slice/crio-a493fc865f8a3184a1ba2a04b035e530aac35efbdfc72fb855ab48a9f84daba0 WatchSource:0}: Error finding container a493fc865f8a3184a1ba2a04b035e530aac35efbdfc72fb855ab48a9f84daba0: Status 404 returned error can't find the container with id a493fc865f8a3184a1ba2a04b035e530aac35efbdfc72fb855ab48a9f84daba0
Oct 30 16:32:39.056671 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:39.056632    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" event={"ID":"5483fdd8-8671-40ed-b785-e54e2a0548d1","Type":"ContainerStarted","Data":"35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91"}
Oct 30 16:32:39.059803 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:39.059774    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn" event={"ID":"d8a7e1af-b485-4226-86c3-fa1bc739c14b","Type":"ContainerStarted","Data":"a493fc865f8a3184a1ba2a04b035e530aac35efbdfc72fb855ab48a9f84daba0"}
Oct 30 16:32:39.627501 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:39.627466    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:39.627820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:39.627796    4497 patch_prober.go:29] interesting pod/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 16:32:39.627886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:39.627846    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 16:32:40.626971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:40.626929    4497 patch_prober.go:29] interesting pod/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 16:32:40.627445 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:40.626986    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 16:32:41.113002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.112956    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"]
Oct 30 16:32:41.116349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.116328    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.137893 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.137849    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 16:32:41.137893 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.137872    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\""
Oct 30 16:32:41.144227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.143839    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"]
Oct 30 16:32:41.169480 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.169445    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.169680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.169524    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.169680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.169604    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlbvt\" (UniqueName: \"kubernetes.io/projected/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-kube-api-access-qlbvt\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.270447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.270413    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlbvt\" (UniqueName: \"kubernetes.io/projected/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-kube-api-access-qlbvt\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.270631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.270524    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.270631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.270607    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.300246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.271426    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.305521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.305448    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.306243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.306182    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlbvt\" (UniqueName: \"kubernetes.io/projected/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-kube-api-access-qlbvt\") pod \"maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.432255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.432149    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:41.627796 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.627765    4497 patch_prober.go:29] interesting pod/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 16:32:41.628089 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.627820    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 16:32:41.785125 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:41.785085    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"]
Oct 30 16:32:42.105406 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:42.105371    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn" event={"ID":"d8a7e1af-b485-4226-86c3-fa1bc739c14b","Type":"ContainerStarted","Data":"111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08"}
Oct 30 16:32:42.105605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:42.105490    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:42.106790 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:42.106767    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t" event={"ID":"b10ca367-c9b5-463e-bfd1-f93d8f4f572c","Type":"ContainerStarted","Data":"74493ba965579421c78731f714f801b779f3beb7ac296c17948447f041d534f4"}
Oct 30 16:32:42.147942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:42.147883    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn" podStartSLOduration=1.934996043 podStartE2EDuration="5.147868362s" totalImagesPullingTime="3.212872319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:42.146279461 +0000 UTC m=+592.000157725" watchObservedRunningTime="2026-10-30 16:32:42.147868362 +0000 UTC m=+592.001746627"
Oct 30 16:32:42.627705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:42.627671    4497 patch_prober.go:29] interesting pod/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 16:32:42.627878 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:42.627726    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 16:32:43.627710 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:43.627621    4497 patch_prober.go:29] interesting pod/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 16:32:43.627710 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:43.627687    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 16:32:44.122584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.122410    4497 generic.go:356] "Generic (PLEG): container finished" podID="63e20db3-a768-4ba9-8588-24043f52043f" containerID="f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5" exitCode=0
Oct 30 16:32:44.122584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.122469    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" event={"ID":"63e20db3-a768-4ba9-8588-24043f52043f","Type":"ContainerDied","Data":"f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5"}
Oct 30 16:32:44.122920 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.122893    4497 scope.go:122] "RemoveContainer" containerID="f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5"
Oct 30 16:32:44.124319 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.124274    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:44.124427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.124358    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:44.124427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.124397    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:44.631844 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.631812    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:44.632381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.632083    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:44.633300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:44.633278    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:32:45.137943 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:45.137907    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" event={"ID":"63e20db3-a768-4ba9-8588-24043f52043f","Type":"ContainerStarted","Data":"79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc"}
Oct 30 16:32:45.140040 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:45.140006    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t" event={"ID":"b10ca367-c9b5-463e-bfd1-f93d8f4f572c","Type":"ContainerStarted","Data":"ca64db6b529d688c008d3244261b9cd770576cf5c1d440be040ee2031f06dd9e"}
Oct 30 16:32:45.238458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:45.238399    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t" podStartSLOduration=1.236926938 podStartE2EDuration="4.238383194s" totalImagesPullingTime="3.001456256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:45.236337988 +0000 UTC m=+595.090216254" watchObservedRunningTime="2026-10-30 16:32:45.238383194 +0000 UTC m=+595.092261460"
Oct 30 16:32:45.503389 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:45.503306    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:45.504647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:45.504619    4497 patch_prober.go:29] interesting pod/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 16:32:45.504791 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:45.504680    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 16:32:46.146837 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:46.146749    4497 generic.go:356] "Generic (PLEG): container finished" podID="b10ca367-c9b5-463e-bfd1-f93d8f4f572c" containerID="ca64db6b529d688c008d3244261b9cd770576cf5c1d440be040ee2031f06dd9e" exitCode=0
Oct 30 16:32:46.147302 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:46.146831    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t" event={"ID":"b10ca367-c9b5-463e-bfd1-f93d8f4f572c","Type":"ContainerDied","Data":"ca64db6b529d688c008d3244261b9cd770576cf5c1d440be040ee2031f06dd9e"}
Oct 30 16:32:46.503994 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:46.503960    4497 patch_prober.go:29] interesting pod/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 16:32:46.504141 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:46.504015    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 16:32:47.329617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.329592    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:47.361072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.361039    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-openshift-ca\") pod \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") "
Oct 30 16:32:47.361263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.361154    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-xks-ca\") pod \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") "
Oct 30 16:32:47.361263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.361250    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-kube-api-access-qlbvt\" (UniqueName: \"kubernetes.io/projected/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-kube-api-access-qlbvt\") pod \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\" (UID: \"b10ca367-c9b5-463e-bfd1-f93d8f4f572c\") "
Oct 30 16:32:47.361473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.361448    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-openshift-ca" pod "b10ca367-c9b5-463e-bfd1-f93d8f4f572c" (UID: "b10ca367-c9b5-463e-bfd1-f93d8f4f572c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:32:47.361562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.361548    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-openshift-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:47.363341 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.363320    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-xks-ca" pod "b10ca367-c9b5-463e-bfd1-f93d8f4f572c" (UID: "b10ca367-c9b5-463e-bfd1-f93d8f4f572c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:32:47.363424 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.363401    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-kube-api-access-qlbvt" pod "b10ca367-c9b5-463e-bfd1-f93d8f4f572c" (UID: "b10ca367-c9b5-463e-bfd1-f93d8f4f572c"). InnerVolumeSpecName "kube-api-access-qlbvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:32:47.463120 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.463028    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlbvt\" (UniqueName: \"kubernetes.io/projected/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-kube-api-access-qlbvt\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:47.463120 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.463061    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b10ca367-c9b5-463e-bfd1-f93d8f4f572c-maas-api-xks-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:47.503958 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.503922    4497 patch_prober.go:29] interesting pod/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 16:32:47.504129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.503978    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 16:32:47.915323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:47.915286    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:48.119107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:48.119078    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:32:48.159392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:48.159364    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"
Oct 30 16:32:48.159584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:48.159411    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t" event={"ID":"b10ca367-c9b5-463e-bfd1-f93d8f4f572c","Type":"ContainerDied","Data":"74493ba965579421c78731f714f801b779f3beb7ac296c17948447f041d534f4"}
Oct 30 16:32:48.159584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:48.159454    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74493ba965579421c78731f714f801b779f3beb7ac296c17948447f041d534f4"
Oct 30 16:32:48.504398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:48.504361    4497 patch_prober.go:29] interesting pod/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 16:32:48.504796 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:48.504426    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 16:32:49.503456 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:49.503425    4497 patch_prober.go:29] interesting pod/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 16:32:49.503642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:49.503472    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 16:32:49.649469 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:49.649438    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"]
Oct 30 16:32:49.699685 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:49.699654    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"]
Oct 30 16:32:50.507691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:50.507662    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:50.508712 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:50.508690    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:32:51.784823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:51.784787    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"]
Oct 30 16:32:51.815637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:51.815601    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"]
Oct 30 16:32:51.916052 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:32:51.916017    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:32:54.651234 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:54.651180    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm" podUID="244468e3-f44c-4055-af6f-3f6474547ea0" containerName="payload-processing" containerID="cri-o://e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643" gracePeriod=25
Oct 30 16:32:54.700277 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:54.700227    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl" podUID="bc72be2b-3fd6-4e02-a30e-4154656cb651" containerName="payload-pre-processing" containerID="cri-o://83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6" gracePeriod=25
Oct 30 16:32:54.952070 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:54.952044    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:54.977626 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:54.977602    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:55.064423 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.064394    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc72be2b-3fd6-4e02-a30e-4154656cb651-kube-api-access-2tzrm\" (UniqueName: \"kubernetes.io/projected/bc72be2b-3fd6-4e02-a30e-4154656cb651-kube-api-access-2tzrm\") pod \"bc72be2b-3fd6-4e02-a30e-4154656cb651\" (UID: \"bc72be2b-3fd6-4e02-a30e-4154656cb651\") "
Oct 30 16:32:55.064593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.064439    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc72be2b-3fd6-4e02-a30e-4154656cb651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc72be2b-3fd6-4e02-a30e-4154656cb651-plugins-config-volume\") pod \"bc72be2b-3fd6-4e02-a30e-4154656cb651\" (UID: \"bc72be2b-3fd6-4e02-a30e-4154656cb651\") "
Oct 30 16:32:55.064593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.064547    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244468e3-f44c-4055-af6f-3f6474547ea0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244468e3-f44c-4055-af6f-3f6474547ea0-plugins-config-volume\") pod \"244468e3-f44c-4055-af6f-3f6474547ea0\" (UID: \"244468e3-f44c-4055-af6f-3f6474547ea0\") "
Oct 30 16:32:55.064672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.064601    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244468e3-f44c-4055-af6f-3f6474547ea0-kube-api-access-zs5xq\" (UniqueName: \"kubernetes.io/projected/244468e3-f44c-4055-af6f-3f6474547ea0-kube-api-access-zs5xq\") pod \"244468e3-f44c-4055-af6f-3f6474547ea0\" (UID: \"244468e3-f44c-4055-af6f-3f6474547ea0\") "
Oct 30 16:32:55.064910 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.064879    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc72be2b-3fd6-4e02-a30e-4154656cb651-plugins-config-volume" pod "bc72be2b-3fd6-4e02-a30e-4154656cb651" (UID: "bc72be2b-3fd6-4e02-a30e-4154656cb651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:32:55.065037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.064970    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244468e3-f44c-4055-af6f-3f6474547ea0-plugins-config-volume" pod "244468e3-f44c-4055-af6f-3f6474547ea0" (UID: "244468e3-f44c-4055-af6f-3f6474547ea0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:32:55.065037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.064987    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc72be2b-3fd6-4e02-a30e-4154656cb651-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:55.066804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.066778    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244468e3-f44c-4055-af6f-3f6474547ea0-kube-api-access-zs5xq" pod "244468e3-f44c-4055-af6f-3f6474547ea0" (UID: "244468e3-f44c-4055-af6f-3f6474547ea0"). InnerVolumeSpecName "kube-api-access-zs5xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:32:55.066804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.066789    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc72be2b-3fd6-4e02-a30e-4154656cb651-kube-api-access-2tzrm" pod "bc72be2b-3fd6-4e02-a30e-4154656cb651" (UID: "bc72be2b-3fd6-4e02-a30e-4154656cb651"). InnerVolumeSpecName "kube-api-access-2tzrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:32:55.166583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.166542    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tzrm\" (UniqueName: \"kubernetes.io/projected/bc72be2b-3fd6-4e02-a30e-4154656cb651-kube-api-access-2tzrm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:55.166583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.166576    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244468e3-f44c-4055-af6f-3f6474547ea0-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:55.166583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.166585    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs5xq\" (UniqueName: \"kubernetes.io/projected/244468e3-f44c-4055-af6f-3f6474547ea0-kube-api-access-zs5xq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:55.199264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.199159    4497 generic.go:356] "Generic (PLEG): container finished" podID="244468e3-f44c-4055-af6f-3f6474547ea0" containerID="e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643" exitCode=0
Oct 30 16:32:55.199264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.199223    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm" event={"ID":"244468e3-f44c-4055-af6f-3f6474547ea0","Type":"ContainerDied","Data":"e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643"}
Oct 30 16:32:55.199264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.199236    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"
Oct 30 16:32:55.199541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.199282    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm" event={"ID":"244468e3-f44c-4055-af6f-3f6474547ea0","Type":"ContainerDied","Data":"f938343baf6a4eca5c29a9bec69d5679cef605399ea041749a4072e35cfcb7c7"}
Oct 30 16:32:55.199541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.199314    4497 scope.go:122] "RemoveContainer" containerID="e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643"
Oct 30 16:32:55.200612 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.200589    4497 generic.go:356] "Generic (PLEG): container finished" podID="bc72be2b-3fd6-4e02-a30e-4154656cb651" containerID="83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6" exitCode=0
Oct 30 16:32:55.200706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.200645    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"
Oct 30 16:32:55.200706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.200677    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl" event={"ID":"bc72be2b-3fd6-4e02-a30e-4154656cb651","Type":"ContainerDied","Data":"83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6"}
Oct 30 16:32:55.200833 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.200730    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl" event={"ID":"bc72be2b-3fd6-4e02-a30e-4154656cb651","Type":"ContainerDied","Data":"19b4ce15de2d3d05be409b28d2798a3952c72fe66fe97f27a1fd741cac7e294b"}
Oct 30 16:32:55.209855 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.209833    4497 scope.go:122] "RemoveContainer" containerID="e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643"
Oct 30 16:32:55.210225 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:32:55.210197    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643\": container with ID starting with e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643 not found: ID does not exist" containerID="e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643"
Oct 30 16:32:55.210337 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.210234    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643"} err="failed to get container status \"e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643\": rpc error: code = NotFound desc = could not find container \"e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643\": container with ID starting with e4778817a1f2c3399d23ceba822d7ca2191ab22ef79bc146aef8628713a26643 not found: ID does not exist"
Oct 30 16:32:55.210337 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.210256    4497 scope.go:122] "RemoveContainer" containerID="83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6"
Oct 30 16:32:55.218617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.218599    4497 scope.go:122] "RemoveContainer" containerID="83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6"
Oct 30 16:32:55.218875 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:32:55.218853    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6\": container with ID starting with 83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6 not found: ID does not exist" containerID="83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6"
Oct 30 16:32:55.218943 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.218885    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6"} err="failed to get container status \"83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6\": rpc error: code = NotFound desc = could not find container \"83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6\": container with ID starting with 83fb94f2084cb5b8c1dfa814f2b82eea3ec41ce9f7457879b27e351ae18362c6 not found: ID does not exist"
Oct 30 16:32:55.243593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.243562    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"]
Oct 30 16:32:55.245704 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.245677    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f756b7bf-69b457c9f-6kwcl"]
Oct 30 16:32:55.275392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.275345    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"]
Oct 30 16:32:55.278737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.278708    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f756b7bf-cdb559648-ts4hm"]
Oct 30 16:32:55.399819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.399785    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"]
Oct 30 16:32:55.444762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:55.444725    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"]
Oct 30 16:32:56.298410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:56.298380    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244468e3-f44c-4055-af6f-3f6474547ea0" path="/var/lib/kubelet/pods/244468e3-f44c-4055-af6f-3f6474547ea0/volumes"
Oct 30 16:32:56.298794 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:56.298762    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc72be2b-3fd6-4e02-a30e-4154656cb651" path="/var/lib/kubelet/pods/bc72be2b-3fd6-4e02-a30e-4154656cb651/volumes"
Oct 30 16:32:56.454209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:56.449022    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"]
Oct 30 16:32:56.454209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:56.449290    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy" containerID="cri-o://35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91" gracePeriod=30
Oct 30 16:32:56.670215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:56.670115    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"]
Oct 30 16:32:56.674245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:56.674219    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a2d47c16-9gp4t"]
Oct 30 16:32:56.785574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:56.785525    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6" podUID="f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" containerName="payload-processing" containerID="cri-o://22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924" gracePeriod=25
Oct 30 16:32:56.816201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:56.816155    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r" podUID="34849fd3-9153-4a70-85d6-a0bfd562625f" containerName="payload-pre-processing" containerID="cri-o://6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683" gracePeriod=25
Oct 30 16:32:56.981018 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:56.980990    4497 logging.go:55] [core] [Channel #59 SubChannel #63]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.45:50052", ServerName: "10.133.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.45:50052: connect: connection refused"
Oct 30 16:32:57.062669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.062644    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:57.099500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.099471    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:57.190146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.190016    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-kube-api-access-dg24g\" (UniqueName: \"kubernetes.io/projected/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-kube-api-access-dg24g\") pod \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\" (UID: \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\") "
Oct 30 16:32:57.190146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.190090    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-plugins-config-volume\") pod \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\" (UID: \"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac\") "
Oct 30 16:32:57.190554 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.190281    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34849fd3-9153-4a70-85d6-a0bfd562625f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34849fd3-9153-4a70-85d6-a0bfd562625f-plugins-config-volume\") pod \"34849fd3-9153-4a70-85d6-a0bfd562625f\" (UID: \"34849fd3-9153-4a70-85d6-a0bfd562625f\") "
Oct 30 16:32:57.190554 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.190389    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34849fd3-9153-4a70-85d6-a0bfd562625f-kube-api-access-wr7hh\" (UniqueName: \"kubernetes.io/projected/34849fd3-9153-4a70-85d6-a0bfd562625f-kube-api-access-wr7hh\") pod \"34849fd3-9153-4a70-85d6-a0bfd562625f\" (UID: \"34849fd3-9153-4a70-85d6-a0bfd562625f\") "
Oct 30 16:32:57.190709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.190646    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-plugins-config-volume" pod "f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" (UID: "f40a2f74-c18c-4c9a-b726-2a20da7aa4ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:32:57.190846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.190827    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:57.191343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.191319    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34849fd3-9153-4a70-85d6-a0bfd562625f-plugins-config-volume" pod "34849fd3-9153-4a70-85d6-a0bfd562625f" (UID: "34849fd3-9153-4a70-85d6-a0bfd562625f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:32:57.192582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.192554    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-kube-api-access-dg24g" pod "f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" (UID: "f40a2f74-c18c-4c9a-b726-2a20da7aa4ac"). InnerVolumeSpecName "kube-api-access-dg24g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:32:57.192711 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.192688    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34849fd3-9153-4a70-85d6-a0bfd562625f-kube-api-access-wr7hh" pod "34849fd3-9153-4a70-85d6-a0bfd562625f" (UID: "34849fd3-9153-4a70-85d6-a0bfd562625f"). InnerVolumeSpecName "kube-api-access-wr7hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:32:57.214923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.214893    4497 generic.go:356] "Generic (PLEG): container finished" podID="f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" containerID="22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924" exitCode=0
Oct 30 16:32:57.215107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.214952    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"
Oct 30 16:32:57.215107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.214977    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6" event={"ID":"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac","Type":"ContainerDied","Data":"22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924"}
Oct 30 16:32:57.215107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.215045    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6" event={"ID":"f40a2f74-c18c-4c9a-b726-2a20da7aa4ac","Type":"ContainerDied","Data":"178b56b2fa575710e02546a6ff492f31bf8f7eddc55d446cec6924f2c20cab3c"}
Oct 30 16:32:57.215107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.215075    4497 scope.go:122] "RemoveContainer" containerID="22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924"
Oct 30 16:32:57.221564 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.221536    4497 generic.go:356] "Generic (PLEG): container finished" podID="34849fd3-9153-4a70-85d6-a0bfd562625f" containerID="6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683" exitCode=0
Oct 30 16:32:57.221718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.221640    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r" event={"ID":"34849fd3-9153-4a70-85d6-a0bfd562625f","Type":"ContainerDied","Data":"6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683"}
Oct 30 16:32:57.221718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.221702    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r" event={"ID":"34849fd3-9153-4a70-85d6-a0bfd562625f","Type":"ContainerDied","Data":"e93ecb6c9c8adab21f5a86ae8a7703cd7a44eab7544a0238cf3711e262401a1a"}
Oct 30 16:32:57.221851 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.221786    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"
Oct 30 16:32:57.234872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.234848    4497 scope.go:122] "RemoveContainer" containerID="22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924"
Oct 30 16:32:57.235231 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:32:57.235209    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924\": container with ID starting with 22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924 not found: ID does not exist" containerID="22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924"
Oct 30 16:32:57.235316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.235238    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924"} err="failed to get container status \"22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924\": rpc error: code = NotFound desc = could not find container \"22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924\": container with ID starting with 22bd6a96735ffb0254dda890998e7be36b8604c15ec168ccb28a24b584062924 not found: ID does not exist"
Oct 30 16:32:57.235316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.235255    4497 scope.go:122] "RemoveContainer" containerID="6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683"
Oct 30 16:32:57.246080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.246063    4497 scope.go:122] "RemoveContainer" containerID="6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683"
Oct 30 16:32:57.246440 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:32:57.246416    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683\": container with ID starting with 6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683 not found: ID does not exist" containerID="6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683"
Oct 30 16:32:57.246536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.246445    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683"} err="failed to get container status \"6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683\": rpc error: code = NotFound desc = could not find container \"6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683\": container with ID starting with 6112b973fdf6c125afbc8269eeb8cb12de3b84bf4851501227f4ca60ba2b5683 not found: ID does not exist"
Oct 30 16:32:57.281174 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.281145    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"]
Oct 30 16:32:57.283296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.283264    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6"]
Oct 30 16:32:57.291521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.291496    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg24g\" (UniqueName: \"kubernetes.io/projected/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac-kube-api-access-dg24g\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:57.291521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.291521    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34849fd3-9153-4a70-85d6-a0bfd562625f-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:57.291698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.291536    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7hh\" (UniqueName: \"kubernetes.io/projected/34849fd3-9153-4a70-85d6-a0bfd562625f-kube-api-access-wr7hh\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:32:57.319100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.319067    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"]
Oct 30 16:32:57.336404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.336370    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r"]
Oct 30 16:32:57.654534 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.654503    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"]
Oct 30 16:32:57.655018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.654999    4497 cpu_manager.go:414] "containerMap: removing container" podUID="bc72be2b-3fd6-4e02-a30e-4154656cb651" containerName="payload-pre-processing"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655019    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc72be2b-3fd6-4e02-a30e-4154656cb651" containerName="payload-pre-processing"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655054    4497 cpu_manager.go:414] "containerMap: removing container" podUID="34849fd3-9153-4a70-85d6-a0bfd562625f" containerName="payload-pre-processing"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655064    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34849fd3-9153-4a70-85d6-a0bfd562625f" containerName="payload-pre-processing"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655092    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" containerName="payload-processing"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655100    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" containerName="payload-processing"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655115    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b10ca367-c9b5-463e-bfd1-f93d8f4f572c" containerName="revoke-keys"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655122    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10ca367-c9b5-463e-bfd1-f93d8f4f572c" containerName="revoke-keys"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655145    4497 cpu_manager.go:414] "containerMap: removing container" podUID="244468e3-f44c-4055-af6f-3f6474547ea0" containerName="payload-processing"
Oct 30 16:32:57.655162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655153    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244468e3-f44c-4055-af6f-3f6474547ea0" containerName="payload-processing"
Oct 30 16:32:57.655647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655295    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" containerName="payload-processing"
Oct 30 16:32:57.655647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655310    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10ca367-c9b5-463e-bfd1-f93d8f4f572c" containerName="revoke-keys"
Oct 30 16:32:57.655647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655323    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="244468e3-f44c-4055-af6f-3f6474547ea0" containerName="payload-processing"
Oct 30 16:32:57.655647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655332    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc72be2b-3fd6-4e02-a30e-4154656cb651" containerName="payload-pre-processing"
Oct 30 16:32:57.655647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.655345    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="34849fd3-9153-4a70-85d6-a0bfd562625f" containerName="payload-pre-processing"
Oct 30 16:32:57.658178 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.658158    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.670662 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.670627    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-18519c-openshift-default-dockercfg-nqftk\""
Oct 30 16:32:57.691323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.691292    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"]
Oct 30 16:32:57.796675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.796639    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.796860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.796681    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xd9p\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-kube-api-access-8xd9p\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.796860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.796702    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.796955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.796850    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.796955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.796891    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.797048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.796961    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019fe5f-39a5-4658-996c-4f91a78231e8-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.797048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.797005    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.797163 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.797139    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.797245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.797216    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.898762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.898717    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019fe5f-39a5-4658-996c-4f91a78231e8-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.898784    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.898853    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.898880    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.898923    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.898958    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xd9p\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-kube-api-access-8xd9p\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.899121    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.899179    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.899261    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.899365    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899557 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.899534    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899621 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.899579    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019fe5f-39a5-4658-996c-4f91a78231e8-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.899630    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.899732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.899697    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.901367 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.901347    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.901496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.901476    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.919715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.919657    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.919810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.919797    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xd9p\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-kube-api-access-8xd9p\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.972515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.972484    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:57.981327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.981287    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-a2d47c16-8668fb4958-lfxt6" podUID="f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.46:50052\" within 1s: context deadline exceeded"
Oct 30 16:32:57.981482 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:57.981365    4497 logging.go:55] [core] [Channel #57 SubChannel #60]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.46:50052", ServerName: "10.133.0.46:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.46:50052: operation was canceled"
Oct 30 16:32:57.981482 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:57.981290    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2d47c16-6954f77b87-tnz9r" podUID="34849fd3-9153-4a70-85d6-a0bfd562625f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.45:50052\" within 1s: context deadline exceeded"
Oct 30 16:32:57.981482 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:57.981372    4497 logging.go:55] [core] [Channel #59 SubChannel #63]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.45:50052", ServerName: "10.133.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.45:50052: operation was canceled"
Oct 30 16:32:58.291601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:58.291564    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"]
Oct 30 16:32:58.295872 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:32:58.295842    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf019fe5f_39a5_4658_996c_4f91a78231e8.slice/crio-44de95dcaf98361d06a4d894ca863124dbcde80e0ae25dba314cfd5608c512ac WatchSource:0}: Error finding container 44de95dcaf98361d06a4d894ca863124dbcde80e0ae25dba314cfd5608c512ac: Status 404 returned error can't find the container with id 44de95dcaf98361d06a4d894ca863124dbcde80e0ae25dba314cfd5608c512ac
Oct 30 16:32:58.298691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:58.298656    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:58.298766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:58.298723    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:58.298766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:58.298752    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:32:58.300323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:58.300294    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34849fd3-9153-4a70-85d6-a0bfd562625f" path="/var/lib/kubelet/pods/34849fd3-9153-4a70-85d6-a0bfd562625f/volumes"
Oct 30 16:32:58.300676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:58.300663    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10ca367-c9b5-463e-bfd1-f93d8f4f572c" path="/var/lib/kubelet/pods/b10ca367-c9b5-463e-bfd1-f93d8f4f572c/volumes"
Oct 30 16:32:58.300989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:58.300977    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40a2f74-c18c-4c9a-b726-2a20da7aa4ac" path="/var/lib/kubelet/pods/f40a2f74-c18c-4c9a-b726-2a20da7aa4ac/volumes"
Oct 30 16:32:59.235361 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:59.235328    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" event={"ID":"f019fe5f-39a5-4658-996c-4f91a78231e8","Type":"ContainerStarted","Data":"846072f3c384abf7d12ce69f59a2c2022e06b9953760f5463b561da350194d11"}
Oct 30 16:32:59.235750 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:59.235418    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" event={"ID":"f019fe5f-39a5-4658-996c-4f91a78231e8","Type":"ContainerStarted","Data":"44de95dcaf98361d06a4d894ca863124dbcde80e0ae25dba314cfd5608c512ac"}
Oct 30 16:32:59.285758 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:59.285695    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" podStartSLOduration=2.28567893 podStartE2EDuration="2.28567893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:59.274869642 +0000 UTC m=+609.128747917" watchObservedRunningTime="2026-10-30 16:32:59.28567893 +0000 UTC m=+609.139557194"
Oct 30 16:32:59.634901 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:59.634866    4497 patch_prober.go:29] interesting pod/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:32:59.635079 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:59.634927    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:32:59.973289 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:59.973201    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:32:59.978238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:32:59.978206    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:33:00.241119 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.241038    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:33:00.242121 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.242104    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:33:00.401137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.401086    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn" podUID="5e980934-b9c1-4cf6-b93f-b4397dbe8753" containerName="payload-processing" containerID="cri-o://fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d" gracePeriod=25
Oct 30 16:33:00.445975 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.445929    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9" podUID="548f988e-1e4a-4017-a5b3-0377cb8d7c7b" containerName="payload-pre-processing" containerID="cri-o://7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17" gracePeriod=25
Oct 30 16:33:00.680602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.680575    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:33:00.737459 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.737434    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:33:00.763976 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.763811    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"]
Oct 30 16:33:00.764156 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.764105    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy" containerID="cri-o://79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc" gracePeriod=30
Oct 30 16:33:00.835407 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.835314    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e980934-b9c1-4cf6-b93f-b4397dbe8753-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e980934-b9c1-4cf6-b93f-b4397dbe8753-plugins-config-volume\") pod \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\" (UID: \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\") "
Oct 30 16:33:00.835598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.835412    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e980934-b9c1-4cf6-b93f-b4397dbe8753-kube-api-access-gnb9h\" (UniqueName: \"kubernetes.io/projected/5e980934-b9c1-4cf6-b93f-b4397dbe8753-kube-api-access-gnb9h\") pod \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\" (UID: \"5e980934-b9c1-4cf6-b93f-b4397dbe8753\") "
Oct 30 16:33:00.835598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.835481    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-plugins-config-volume\") pod \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\" (UID: \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\") "
Oct 30 16:33:00.835598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.835518    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-kube-api-access-ttqxm\" (UniqueName: \"kubernetes.io/projected/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-kube-api-access-ttqxm\") pod \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\" (UID: \"548f988e-1e4a-4017-a5b3-0377cb8d7c7b\") "
Oct 30 16:33:00.835883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.835854    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e980934-b9c1-4cf6-b93f-b4397dbe8753-plugins-config-volume" pod "5e980934-b9c1-4cf6-b93f-b4397dbe8753" (UID: "5e980934-b9c1-4cf6-b93f-b4397dbe8753"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:33:00.835947 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.835883    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-plugins-config-volume" pod "548f988e-1e4a-4017-a5b3-0377cb8d7c7b" (UID: "548f988e-1e4a-4017-a5b3-0377cb8d7c7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:33:00.837614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.837590    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-kube-api-access-ttqxm" pod "548f988e-1e4a-4017-a5b3-0377cb8d7c7b" (UID: "548f988e-1e4a-4017-a5b3-0377cb8d7c7b"). InnerVolumeSpecName "kube-api-access-ttqxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:33:00.837614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.837611    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e980934-b9c1-4cf6-b93f-b4397dbe8753-kube-api-access-gnb9h" pod "5e980934-b9c1-4cf6-b93f-b4397dbe8753" (UID: "5e980934-b9c1-4cf6-b93f-b4397dbe8753"). InnerVolumeSpecName "kube-api-access-gnb9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:33:00.936398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.936366    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:00.936398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.936395    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttqxm\" (UniqueName: \"kubernetes.io/projected/548f988e-1e4a-4017-a5b3-0377cb8d7c7b-kube-api-access-ttqxm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:00.936398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.936405    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e980934-b9c1-4cf6-b93f-b4397dbe8753-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:00.936398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:00.936413    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnb9h\" (UniqueName: \"kubernetes.io/projected/5e980934-b9c1-4cf6-b93f-b4397dbe8753-kube-api-access-gnb9h\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.246741 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.246652    4497 generic.go:356] "Generic (PLEG): container finished" podID="548f988e-1e4a-4017-a5b3-0377cb8d7c7b" containerID="7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17" exitCode=0
Oct 30 16:33:01.246741 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.246694    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"
Oct 30 16:33:01.246741 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.246731    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9" event={"ID":"548f988e-1e4a-4017-a5b3-0377cb8d7c7b","Type":"ContainerDied","Data":"7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17"}
Oct 30 16:33:01.247381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.246783    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9" event={"ID":"548f988e-1e4a-4017-a5b3-0377cb8d7c7b","Type":"ContainerDied","Data":"58cdd760354cf1c10a119a878a14f3c8c73794db807459b209b6091118491f50"}
Oct 30 16:33:01.247381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.246808    4497 scope.go:122] "RemoveContainer" containerID="7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17"
Oct 30 16:33:01.248086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.248061    4497 generic.go:356] "Generic (PLEG): container finished" podID="5e980934-b9c1-4cf6-b93f-b4397dbe8753" containerID="fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d" exitCode=0
Oct 30 16:33:01.248169 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.248091    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn" event={"ID":"5e980934-b9c1-4cf6-b93f-b4397dbe8753","Type":"ContainerDied","Data":"fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d"}
Oct 30 16:33:01.248169 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.248105    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"
Oct 30 16:33:01.248169 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.248130    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn" event={"ID":"5e980934-b9c1-4cf6-b93f-b4397dbe8753","Type":"ContainerDied","Data":"1fc75a8b3b494ad32fe79779d3f0fd781e798c3643a88a0baeaa5edf57b107e0"}
Oct 30 16:33:01.260929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.260907    4497 scope.go:122] "RemoveContainer" containerID="7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17"
Oct 30 16:33:01.261282 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:01.261253    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17\": container with ID starting with 7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17 not found: ID does not exist" containerID="7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17"
Oct 30 16:33:01.261403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.261292    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17"} err="failed to get container status \"7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17\": rpc error: code = NotFound desc = could not find container \"7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17\": container with ID starting with 7b1ff0d1f386abf835ab09e9e157d59914ca545c87210bce6cbb006b615fda17 not found: ID does not exist"
Oct 30 16:33:01.261403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.261314    4497 scope.go:122] "RemoveContainer" containerID="fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d"
Oct 30 16:33:01.270692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.270668    4497 scope.go:122] "RemoveContainer" containerID="fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d"
Oct 30 16:33:01.270977 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:01.270958    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d\": container with ID starting with fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d not found: ID does not exist" containerID="fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d"
Oct 30 16:33:01.271062 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.270986    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d"} err="failed to get container status \"fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d\": rpc error: code = NotFound desc = could not find container \"fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d\": container with ID starting with fe54f291dbfb9790fb304f88352b0701d9722acc2a8e6a1652012763cd0e623d not found: ID does not exist"
Oct 30 16:33:01.293722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.293683    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"]
Oct 30 16:33:01.299571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.299533    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cff59996-bc4cf7864-gnxpn"]
Oct 30 16:33:01.328574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.328547    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"]
Oct 30 16:33:01.332473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.332450    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cff59996-6c58bdf7f-6ssv9"]
Oct 30 16:33:01.631601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.631578    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:33:01.744163 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744130    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-credential-socket\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744224    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-token\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-token\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744282    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-certs\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744340    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5483fdd8-8671-40ed-b785-e54e2a0548d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483fdd8-8671-40ed-b785-e54e2a0548d1-istiod-ca-cert\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744379    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-kube-api-access-f9775\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-kube-api-access-f9775\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744430    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-socket\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744431    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-credential-socket" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:01.744530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744483    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-envoy\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744527    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-data\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744573    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-podinfo\") pod \"5483fdd8-8671-40ed-b785-e54e2a0548d1\" (UID: \"5483fdd8-8671-40ed-b785-e54e2a0548d1\") "
Oct 30 16:33:01.744781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744587    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-certs" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:01.744781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744726    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-socket" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:01.744781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744735    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5483fdd8-8671-40ed-b785-e54e2a0548d1-istiod-ca-cert" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:33:01.744962 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744929    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.744962 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744952    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.745058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744967    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5483fdd8-8671-40ed-b785-e54e2a0548d1-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.745058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.744980    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.745058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.745000    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-data" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:01.747266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.746697    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-token" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:33:01.747404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.747380    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-envoy" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:01.747503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.747063    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-podinfo" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:33:01.747503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.747122    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-kube-api-access-f9775" pod "5483fdd8-8671-40ed-b785-e54e2a0548d1" (UID: "5483fdd8-8671-40ed-b785-e54e2a0548d1"). InnerVolumeSpecName "kube-api-access-f9775". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:33:01.845541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.845460    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.845541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.845491    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9775\" (UniqueName: \"kubernetes.io/projected/5483fdd8-8671-40ed-b785-e54e2a0548d1-kube-api-access-f9775\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.845541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.845500    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.845541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.845509    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.845541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.845516    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5483fdd8-8671-40ed-b785-e54e2a0548d1-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:01.991530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.991496    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"]
Oct 30 16:33:01.992137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992118    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy"
Oct 30 16:33:01.992247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992139    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy"
Oct 30 16:33:01.992247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992153    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy"
Oct 30 16:33:01.992247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992161    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy"
Oct 30 16:33:01.992247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992219    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5e980934-b9c1-4cf6-b93f-b4397dbe8753" containerName="payload-processing"
Oct 30 16:33:01.992247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992228    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e980934-b9c1-4cf6-b93f-b4397dbe8753" containerName="payload-processing"
Oct 30 16:33:01.992433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992249    4497 cpu_manager.go:414] "containerMap: removing container" podUID="548f988e-1e4a-4017-a5b3-0377cb8d7c7b" containerName="payload-pre-processing"
Oct 30 16:33:01.992433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992257    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548f988e-1e4a-4017-a5b3-0377cb8d7c7b" containerName="payload-pre-processing"
Oct 30 16:33:01.992433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992369    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy"
Oct 30 16:33:01.992433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992382    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerName="istio-proxy"
Oct 30 16:33:01.992433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992397    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e980934-b9c1-4cf6-b93f-b4397dbe8753" containerName="payload-processing"
Oct 30 16:33:01.992433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.992409    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="548f988e-1e4a-4017-a5b3-0377cb8d7c7b" containerName="payload-pre-processing"
Oct 30 16:33:01.995715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:01.995695    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.010784 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.010762    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-420ff3-openshift-default-dockercfg-mlfh4\""
Oct 30 16:33:02.022928 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.022896    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"]
Oct 30 16:33:02.148410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148304    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjnj9\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-kube-api-access-cjnj9\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.148410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148347    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.148410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148370    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.148768 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148580    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.148768 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148619    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.148768 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148701    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.148768 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148748    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.148964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148779    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.148964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.148821    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250426 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250389    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250436    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250479    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250518    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250559    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250596    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250651    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjnj9\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-kube-api-access-cjnj9\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250683    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250715    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250839    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.250952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.250903    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.251496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.251137    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.251496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.251281    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.251496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.251333    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.252981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.252960    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.253463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.253446    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.258882 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.258860    4497 generic.go:356] "Generic (PLEG): container finished" podID="5483fdd8-8671-40ed-b785-e54e2a0548d1" containerID="35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91" exitCode=0
Oct 30 16:33:02.259008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.258940    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"
Oct 30 16:33:02.259008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.258941    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" event={"ID":"5483fdd8-8671-40ed-b785-e54e2a0548d1","Type":"ContainerDied","Data":"35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91"}
Oct 30 16:33:02.259118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.259004    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr" event={"ID":"5483fdd8-8671-40ed-b785-e54e2a0548d1","Type":"ContainerDied","Data":"821eee6c387fc5f861bd5dd3d1b02a208193d2c1180e64b262df0b7889da5911"}
Oct 30 16:33:02.259118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.259036    4497 scope.go:122] "RemoveContainer" containerID="35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91"
Oct 30 16:33:02.271041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.271020    4497 scope.go:122] "RemoveContainer" containerID="c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5"
Oct 30 16:33:02.272144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.272123    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.272411 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.272395    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjnj9\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-kube-api-access-cjnj9\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.279146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.279129    4497 scope.go:122] "RemoveContainer" containerID="35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91"
Oct 30 16:33:02.279414 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:02.279395    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91\": container with ID starting with 35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91 not found: ID does not exist" containerID="35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91"
Oct 30 16:33:02.279488 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.279425    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91"} err="failed to get container status \"35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91\": rpc error: code = NotFound desc = could not find container \"35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91\": container with ID starting with 35ea6b27a15939cc7f97ecbf7623b252f38e2441ea5d55ee5616affc80739d91 not found: ID does not exist"
Oct 30 16:33:02.279488 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.279449    4497 scope.go:122] "RemoveContainer" containerID="c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5"
Oct 30 16:33:02.279681 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:02.279665    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5\": container with ID starting with c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5 not found: ID does not exist" containerID="c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5"
Oct 30 16:33:02.279719 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.279685    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5"} err="failed to get container status \"c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5\": rpc error: code = NotFound desc = could not find container \"c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5\": container with ID starting with c75139995a5f69b70ef482f051b90509e7800fd82ca3076dd493cf14522c58a5 not found: ID does not exist"
Oct 30 16:33:02.298813 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.298787    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548f988e-1e4a-4017-a5b3-0377cb8d7c7b" path="/var/lib/kubelet/pods/548f988e-1e4a-4017-a5b3-0377cb8d7c7b/volumes"
Oct 30 16:33:02.299156 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.299144    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e980934-b9c1-4cf6-b93f-b4397dbe8753" path="/var/lib/kubelet/pods/5e980934-b9c1-4cf6-b93f-b4397dbe8753/volumes"
Oct 30 16:33:02.303599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.303575    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"]
Oct 30 16:33:02.310417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.310394    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f756b7bf-openshift-default-55d498cc9-mbqcr"]
Oct 30 16:33:02.311545 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.311488    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:02.632216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.632164    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"]
Oct 30 16:33:02.635149 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:33:02.635115    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d45ae6_16d3_4605_95ab_19e3b35c4c96.slice/crio-573678f4e8dcb333c315e426a8c0e8927dba3c28b10904f77450b395a447b6dd WatchSource:0}: Error finding container 573678f4e8dcb333c315e426a8c0e8927dba3c28b10904f77450b395a447b6dd: Status 404 returned error can't find the container with id 573678f4e8dcb333c315e426a8c0e8927dba3c28b10904f77450b395a447b6dd
Oct 30 16:33:02.637592 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.637560    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:02.637672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.637624    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:02.637672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:02.637654    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:03.265849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:03.265807    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" event={"ID":"c5d45ae6-16d3-4605-95ab-19e3b35c4c96","Type":"ContainerStarted","Data":"f76a693c91b04fd7917905de51ee79281c802a6013a8b8080b957da62d2445a6"}
Oct 30 16:33:03.265849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:03.265855    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" event={"ID":"c5d45ae6-16d3-4605-95ab-19e3b35c4c96","Type":"ContainerStarted","Data":"573678f4e8dcb333c315e426a8c0e8927dba3c28b10904f77450b395a447b6dd"}
Oct 30 16:33:03.314245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:03.312613    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:03.317507 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:03.317264    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" podStartSLOduration=2.317243564 podStartE2EDuration="2.317243564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:03.313598611 +0000 UTC m=+613.167476877" watchObservedRunningTime="2026-10-30 16:33:03.317243564 +0000 UTC m=+613.171121831"
Oct 30 16:33:03.340719 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:03.340690    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:04.271254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:04.271223    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:04.272331 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:04.272317    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:04.298598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:04.298560    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5483fdd8-8671-40ed-b785-e54e2a0548d1" path="/var/lib/kubelet/pods/5483fdd8-8671-40ed-b785-e54e2a0548d1/volumes"
Oct 30 16:33:05.510932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.510898    4497 patch_prober.go:29] interesting pod/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 16:33:05.511409 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.510956    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:33:05.696058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.696024    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"]
Oct 30 16:33:05.699538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.699518    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:05.726266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.726235    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-18519c\""
Oct 30 16:33:05.726592 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.726567    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-18519c-dockercfg-l8nxz\""
Oct 30 16:33:05.729247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.729226    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"]
Oct 30 16:33:05.786710 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.786524    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twtl\" (UniqueName: \"kubernetes.io/projected/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-kube-api-access-9twtl\") pod \"payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb\" (UID: \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:05.786710 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.786609    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb\" (UID: \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:05.890039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.888159    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twtl\" (UniqueName: \"kubernetes.io/projected/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-kube-api-access-9twtl\") pod \"payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb\" (UID: \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:05.890039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.888259    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb\" (UID: \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:05.890039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.889172    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb\" (UID: \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:05.911838 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.911795    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twtl\" (UniqueName: \"kubernetes.io/projected/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-kube-api-access-9twtl\") pod \"payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb\" (UID: \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:05.936495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.936469    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:33:05.989134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989103    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-envoy\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989140    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63e20db3-a768-4ba9-8588-24043f52043f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e20db3-a768-4ba9-8588-24043f52043f-istio-podinfo\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989376 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989170    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-data\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989376 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989266    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-credential-socket\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989376 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989303    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-socket\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989376 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989335    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-istio-token\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-istio-token\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989383    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-kube-api-access-jwdxv\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-kube-api-access-jwdxv\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989426    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-certs\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989519    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e20db3-a768-4ba9-8588-24043f52043f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e20db3-a768-4ba9-8588-24043f52043f-istiod-ca-cert\") pod \"63e20db3-a768-4ba9-8588-24043f52043f\" (UID: \"63e20db3-a768-4ba9-8588-24043f52043f\") "
Oct 30 16:33:05.989628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989579    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-data" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:05.989628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989590    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-credential-socket" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:05.989905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989822    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-certs" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:05.989905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989846    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-socket" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:05.990007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989912    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:05.990007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989933    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:05.990007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.989947    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:05.990140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.990119    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e20db3-a768-4ba9-8588-24043f52043f-istiod-ca-cert" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:33:05.991825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.991797    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-kube-api-access-jwdxv" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "kube-api-access-jwdxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:33:05.991963 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.991938    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-envoy" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:33:05.992100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.992009    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-istio-token" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:33:05.992233 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:05.992152    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63e20db3-a768-4ba9-8588-24043f52043f-istio-podinfo" pod "63e20db3-a768-4ba9-8588-24043f52043f" (UID: "63e20db3-a768-4ba9-8588-24043f52043f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:33:06.012341 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.012311    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:06.090961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.090920    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:06.090961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.090959    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:06.091159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.090974    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdxv\" (UniqueName: \"kubernetes.io/projected/63e20db3-a768-4ba9-8588-24043f52043f-kube-api-access-jwdxv\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:06.091159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.090990    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e20db3-a768-4ba9-8588-24043f52043f-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:06.091159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.091003    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e20db3-a768-4ba9-8588-24043f52043f-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:06.091159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.091017    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e20db3-a768-4ba9-8588-24043f52043f-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:06.281484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.281446    4497 generic.go:356] "Generic (PLEG): container finished" podID="63e20db3-a768-4ba9-8588-24043f52043f" containerID="79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc" exitCode=0
Oct 30 16:33:06.281646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.281504    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"
Oct 30 16:33:06.281646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.281534    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" event={"ID":"63e20db3-a768-4ba9-8588-24043f52043f","Type":"ContainerDied","Data":"79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc"}
Oct 30 16:33:06.281646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.281623    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4" event={"ID":"63e20db3-a768-4ba9-8588-24043f52043f","Type":"ContainerDied","Data":"51ef29ca8eda0c7f799d3e64d1522c55c3f9445f420a7cf106ac95cd7720d574"}
Oct 30 16:33:06.281646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.281642    4497 scope.go:122] "RemoveContainer" containerID="79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc"
Oct 30 16:33:06.291302 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.291279    4497 scope.go:122] "RemoveContainer" containerID="f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5"
Oct 30 16:33:06.299777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.299758    4497 scope.go:122] "RemoveContainer" containerID="79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc"
Oct 30 16:33:06.300027 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:06.300010    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc\": container with ID starting with 79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc not found: ID does not exist" containerID="79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc"
Oct 30 16:33:06.300067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.300038    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc"} err="failed to get container status \"79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc\": rpc error: code = NotFound desc = could not find container \"79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc\": container with ID starting with 79180e315d674390962e6c6b7c2fdf3966569357560670e60bf560e5142308cc not found: ID does not exist"
Oct 30 16:33:06.300067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.300055    4497 scope.go:122] "RemoveContainer" containerID="f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5"
Oct 30 16:33:06.300328 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:06.300309    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5\": container with ID starting with f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5 not found: ID does not exist" containerID="f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5"
Oct 30 16:33:06.300412 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.300335    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5"} err="failed to get container status \"f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5\": rpc error: code = NotFound desc = could not find container \"f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5\": container with ID starting with f0bce69f0a949eca07c98a31033adc21255b9430c1a706ad9ef9e97d680d49f5 not found: ID does not exist"
Oct 30 16:33:06.320282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.320173    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"]
Oct 30 16:33:06.323973 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:33:06.323937    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef79d9f_ddb1_46f4_a3bb_5376ed4ba300.slice/crio-3f509d86030d3be753f3ffb964d04b64b147183eb77a12a88a8bd33c36b1a085 WatchSource:0}: Error finding container 3f509d86030d3be753f3ffb964d04b64b147183eb77a12a88a8bd33c36b1a085: Status 404 returned error can't find the container with id 3f509d86030d3be753f3ffb964d04b64b147183eb77a12a88a8bd33c36b1a085
Oct 30 16:33:06.334551 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.334523    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"]
Oct 30 16:33:06.342897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.342870    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cff59996-openshift-default-55f88d9fff-572z4"]
Oct 30 16:33:06.578140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.578054    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"]
Oct 30 16:33:06.578671 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.578651    4497 cpu_manager.go:414] "containerMap: removing container" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy"
Oct 30 16:33:06.578746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.578674    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy"
Oct 30 16:33:06.578746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.578702    4497 cpu_manager.go:414] "containerMap: removing container" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy"
Oct 30 16:33:06.578746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.578709    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy"
Oct 30 16:33:06.578881 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.578812    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy"
Oct 30 16:33:06.582158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.582140    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:06.595452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.595427    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9968-b523-4c4d-ae98-5183512a13ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp\" (UID: \"c2eb9968-b523-4c4d-ae98-5183512a13ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:06.595559 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.595496    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4jk\" (UniqueName: \"kubernetes.io/projected/c2eb9968-b523-4c4d-ae98-5183512a13ce-kube-api-access-jb4jk\") pod \"payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp\" (UID: \"c2eb9968-b523-4c4d-ae98-5183512a13ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:06.595814 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.595796    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f3503b\""
Oct 30 16:33:06.595855 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.595814    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f3503b-dockercfg-j4wz4\""
Oct 30 16:33:06.602422 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.602390    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"]
Oct 30 16:33:06.631672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.631630    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"]
Oct 30 16:33:06.632495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.632468    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e20db3-a768-4ba9-8588-24043f52043f" containerName="istio-proxy"
Oct 30 16:33:06.635834 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.635811    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:06.657498 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.657426    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"]
Oct 30 16:33:06.696651 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.696617    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713416-d816-4724-bfd3-88ad575d63f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l\" (UID: \"ed713416-d816-4724-bfd3-88ad575d63f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:06.696864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.696688    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4jk\" (UniqueName: \"kubernetes.io/projected/c2eb9968-b523-4c4d-ae98-5183512a13ce-kube-api-access-jb4jk\") pod \"payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp\" (UID: \"c2eb9968-b523-4c4d-ae98-5183512a13ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:06.696864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.696756    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbv2\" (UniqueName: \"kubernetes.io/projected/ed713416-d816-4724-bfd3-88ad575d63f2-kube-api-access-ngbv2\") pod \"payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l\" (UID: \"ed713416-d816-4724-bfd3-88ad575d63f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:06.696864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.696821    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9968-b523-4c4d-ae98-5183512a13ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp\" (UID: \"c2eb9968-b523-4c4d-ae98-5183512a13ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:06.697699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.697673    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9968-b523-4c4d-ae98-5183512a13ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp\" (UID: \"c2eb9968-b523-4c4d-ae98-5183512a13ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:06.718787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.718749    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4jk\" (UniqueName: \"kubernetes.io/projected/c2eb9968-b523-4c4d-ae98-5183512a13ce-kube-api-access-jb4jk\") pod \"payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp\" (UID: \"c2eb9968-b523-4c4d-ae98-5183512a13ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:06.798063 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.798012    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713416-d816-4724-bfd3-88ad575d63f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l\" (UID: \"ed713416-d816-4724-bfd3-88ad575d63f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:06.798314 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.798137    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbv2\" (UniqueName: \"kubernetes.io/projected/ed713416-d816-4724-bfd3-88ad575d63f2-kube-api-access-ngbv2\") pod \"payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l\" (UID: \"ed713416-d816-4724-bfd3-88ad575d63f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:06.798980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.798961    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713416-d816-4724-bfd3-88ad575d63f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l\" (UID: \"ed713416-d816-4724-bfd3-88ad575d63f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:06.819162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.819133    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbv2\" (UniqueName: \"kubernetes.io/projected/ed713416-d816-4724-bfd3-88ad575d63f2-kube-api-access-ngbv2\") pod \"payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l\" (UID: \"ed713416-d816-4724-bfd3-88ad575d63f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:06.894731 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.894638    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:06.949075 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:06.949049    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:07.196039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.195951    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"]
Oct 30 16:33:07.200731 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:33:07.200699    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2eb9968_b523_4c4d_ae98_5183512a13ce.slice/crio-503512f063e32d3dfdb6892d1457c58baac072eafd846a36a09f1f272a4c14cd WatchSource:0}: Error finding container 503512f063e32d3dfdb6892d1457c58baac072eafd846a36a09f1f272a4c14cd: Status 404 returned error can't find the container with id 503512f063e32d3dfdb6892d1457c58baac072eafd846a36a09f1f272a4c14cd
Oct 30 16:33:07.265565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.265536    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"]
Oct 30 16:33:07.270987 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:33:07.270961    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded713416_d816_4724_bfd3_88ad575d63f2.slice/crio-aba76442fc168017ef5fb6a9faac68c2bd81f9ca8f0ffcbb572f690d23bf4360 WatchSource:0}: Error finding container aba76442fc168017ef5fb6a9faac68c2bd81f9ca8f0ffcbb572f690d23bf4360: Status 404 returned error can't find the container with id aba76442fc168017ef5fb6a9faac68c2bd81f9ca8f0ffcbb572f690d23bf4360
Oct 30 16:33:07.289588 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.289554    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb" event={"ID":"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300","Type":"ContainerStarted","Data":"6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f"}
Oct 30 16:33:07.289721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.289614    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb" event={"ID":"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300","Type":"ContainerStarted","Data":"3f509d86030d3be753f3ffb964d04b64b147183eb77a12a88a8bd33c36b1a085"}
Oct 30 16:33:07.289721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.289719    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:07.291490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.291467    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l" event={"ID":"ed713416-d816-4724-bfd3-88ad575d63f2","Type":"ContainerStarted","Data":"aba76442fc168017ef5fb6a9faac68c2bd81f9ca8f0ffcbb572f690d23bf4360"}
Oct 30 16:33:07.293227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.293170    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp" event={"ID":"c2eb9968-b523-4c4d-ae98-5183512a13ce","Type":"ContainerStarted","Data":"0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51"}
Oct 30 16:33:07.293344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.293255    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:07.293344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.293279    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp" event={"ID":"c2eb9968-b523-4c4d-ae98-5183512a13ce","Type":"ContainerStarted","Data":"503512f063e32d3dfdb6892d1457c58baac072eafd846a36a09f1f272a4c14cd"}
Oct 30 16:33:07.327942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.327899    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb" podStartSLOduration=2.327883934 podStartE2EDuration="2.327883934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:07.326471963 +0000 UTC m=+617.180350229" watchObservedRunningTime="2026-10-30 16:33:07.327883934 +0000 UTC m=+617.181762199"
Oct 30 16:33:07.370067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.370016    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp" podStartSLOduration=1.369997044 podStartE2EDuration="1.369997044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:07.369683931 +0000 UTC m=+617.223562196" watchObservedRunningTime="2026-10-30 16:33:07.369997044 +0000 UTC m=+617.223875310"
Oct 30 16:33:07.541559 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.541497    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"]
Oct 30 16:33:07.545789 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.545766    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.557724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.557697    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-18519c-serving-cert\""
Oct 30 16:33:07.569549 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.569522    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"]
Oct 30 16:33:07.606076 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.606043    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-metrics-certs\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.606486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.606097    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-maas-api-tls\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.606486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.606114    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/918e1718-9153-44eb-8b8b-f26252675214-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.606486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.606290    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6wg\" (UniqueName: \"kubernetes.io/projected/918e1718-9153-44eb-8b8b-f26252675214-kube-api-access-vm6wg\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.707780 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.707745    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-metrics-certs\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.707969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.707803    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-maas-api-tls\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.707969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.707920    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/918e1718-9153-44eb-8b8b-f26252675214-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.708040 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.708006    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6wg\" (UniqueName: \"kubernetes.io/projected/918e1718-9153-44eb-8b8b-f26252675214-kube-api-access-vm6wg\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.708537 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.708508    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/918e1718-9153-44eb-8b8b-f26252675214-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.710348 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.710328    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-maas-api-tls\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.710437 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.710360    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-metrics-certs\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.728735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.728702    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6wg\" (UniqueName: \"kubernetes.io/projected/918e1718-9153-44eb-8b8b-f26252675214-kube-api-access-vm6wg\") pod \"maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:07.858081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:07.857983    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:08.158296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:08.158065    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"]
Oct 30 16:33:08.162725 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:33:08.162696    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918e1718_9153_44eb_8b8b_f26252675214.slice/crio-b692d7ed26660902d658d63769c3fd33eabaa8a92e166e5d80429fa8e211188a WatchSource:0}: Error finding container b692d7ed26660902d658d63769c3fd33eabaa8a92e166e5d80429fa8e211188a: Status 404 returned error can't find the container with id b692d7ed26660902d658d63769c3fd33eabaa8a92e166e5d80429fa8e211188a
Oct 30 16:33:08.164279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:08.164261    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:33:08.312284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:08.311818    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e20db3-a768-4ba9-8588-24043f52043f" path="/var/lib/kubelet/pods/63e20db3-a768-4ba9-8588-24043f52043f/volumes"
Oct 30 16:33:08.318649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:08.318607    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l" event={"ID":"ed713416-d816-4724-bfd3-88ad575d63f2","Type":"ContainerStarted","Data":"9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0"}
Oct 30 16:33:08.318800 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:08.318713    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:08.319980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:08.319952    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt" event={"ID":"918e1718-9153-44eb-8b8b-f26252675214","Type":"ContainerStarted","Data":"b692d7ed26660902d658d63769c3fd33eabaa8a92e166e5d80429fa8e211188a"}
Oct 30 16:33:08.403570 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:08.403508    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l" podStartSLOduration=2.403490274 podStartE2EDuration="2.403490274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:08.401321971 +0000 UTC m=+618.255200251" watchObservedRunningTime="2026-10-30 16:33:08.403490274 +0000 UTC m=+618.257368539"
Oct 30 16:33:09.327673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.327626    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt" event={"ID":"918e1718-9153-44eb-8b8b-f26252675214","Type":"ContainerStarted","Data":"dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2"}
Oct 30 16:33:09.328084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.327960    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:09.368432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.368378    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt" podStartSLOduration=1.988199792 podStartE2EDuration="2.368364123s" totalImagesPullingTime="380.164331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:09.366708961 +0000 UTC m=+619.220587227" watchObservedRunningTime="2026-10-30 16:33:09.368364123 +0000 UTC m=+619.222242388"
Oct 30 16:33:09.452441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.452285    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"]
Oct 30 16:33:09.452605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.452546    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn" podUID="d8a7e1af-b485-4226-86c3-fa1bc739c14b" containerName="maas-api" containerID="cri-o://111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08" gracePeriod=30
Oct 30 16:33:09.719055 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.719028    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:33:09.831934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.831882    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls\") pod \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") "
Oct 30 16:33:09.831934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.831942    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-metrics-certs\") pod \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") "
Oct 30 16:33:09.832222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.832030    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a7e1af-b485-4226-86c3-fa1bc739c14b-kube-api-access-z5bjz\" (UniqueName: \"kubernetes.io/projected/d8a7e1af-b485-4226-86c3-fa1bc739c14b-kube-api-access-z5bjz\") pod \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") "
Oct 30 16:33:09.832222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.832068    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a7e1af-b485-4226-86c3-fa1bc739c14b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a7e1af-b485-4226-86c3-fa1bc739c14b-openshift-service-ca\") pod \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\" (UID: \"d8a7e1af-b485-4226-86c3-fa1bc739c14b\") "
Oct 30 16:33:09.832507 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.832474    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a7e1af-b485-4226-86c3-fa1bc739c14b-openshift-service-ca" pod "d8a7e1af-b485-4226-86c3-fa1bc739c14b" (UID: "d8a7e1af-b485-4226-86c3-fa1bc739c14b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:33:09.834231 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.834203    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-metrics-certs" pod "d8a7e1af-b485-4226-86c3-fa1bc739c14b" (UID: "d8a7e1af-b485-4226-86c3-fa1bc739c14b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:33:09.834326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.834232    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a7e1af-b485-4226-86c3-fa1bc739c14b-kube-api-access-z5bjz" pod "d8a7e1af-b485-4226-86c3-fa1bc739c14b" (UID: "d8a7e1af-b485-4226-86c3-fa1bc739c14b"). InnerVolumeSpecName "kube-api-access-z5bjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:33:09.834326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.834249    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls" pod "d8a7e1af-b485-4226-86c3-fa1bc739c14b" (UID: "d8a7e1af-b485-4226-86c3-fa1bc739c14b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:33:09.933463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.933369    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5bjz\" (UniqueName: \"kubernetes.io/projected/d8a7e1af-b485-4226-86c3-fa1bc739c14b-kube-api-access-z5bjz\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:09.933463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.933412    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a7e1af-b485-4226-86c3-fa1bc739c14b-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:09.933463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.933422    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:09.933463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:09.933431    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a7e1af-b485-4226-86c3-fa1bc739c14b-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:33:10.139630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.139593    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"]
Oct 30 16:33:10.140104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.140075    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d8a7e1af-b485-4226-86c3-fa1bc739c14b" containerName="maas-api"
Oct 30 16:33:10.140104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.140093    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a7e1af-b485-4226-86c3-fa1bc739c14b" containerName="maas-api"
Oct 30 16:33:10.140258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.140178    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a7e1af-b485-4226-86c3-fa1bc739c14b" containerName="maas-api"
Oct 30 16:33:10.143458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.143440    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.155622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.155599    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f3503b-serving-cert\""
Oct 30 16:33:10.168023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.167995    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"]
Oct 30 16:33:10.236643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.236552    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.236643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.236589    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.236643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.236630    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.236897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.236742    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4xr\" (UniqueName: \"kubernetes.io/projected/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-kube-api-access-vf4xr\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.335009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.334978    4497 generic.go:356] "Generic (PLEG): container finished" podID="d8a7e1af-b485-4226-86c3-fa1bc739c14b" containerID="111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08" exitCode=0
Oct 30 16:33:10.335421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.335021    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"
Oct 30 16:33:10.335421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.335062    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn" event={"ID":"d8a7e1af-b485-4226-86c3-fa1bc739c14b","Type":"ContainerDied","Data":"111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08"}
Oct 30 16:33:10.335421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.335115    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn" event={"ID":"d8a7e1af-b485-4226-86c3-fa1bc739c14b","Type":"ContainerDied","Data":"a493fc865f8a3184a1ba2a04b035e530aac35efbdfc72fb855ab48a9f84daba0"}
Oct 30 16:33:10.335421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.335134    4497 scope.go:122] "RemoveContainer" containerID="111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08"
Oct 30 16:33:10.337421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.337393    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.337548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.337457    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4xr\" (UniqueName: \"kubernetes.io/projected/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-kube-api-access-vf4xr\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.337655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.337636    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.337725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.337670    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.338454 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.338424    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.339993 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.339973    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.340299 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.340283    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.346772 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.346648    4497 scope.go:122] "RemoveContainer" containerID="111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08"
Oct 30 16:33:10.351212 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:10.348994    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08\": container with ID starting with 111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08 not found: ID does not exist" containerID="111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08"
Oct 30 16:33:10.351212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.349042    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08"} err="failed to get container status \"111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08\": rpc error: code = NotFound desc = could not find container \"111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08\": container with ID starting with 111ea5f460c041ae67e1202e0b806a4aebb09b7d5f5868d98b8087d3ee0e3a08 not found: ID does not exist"
Oct 30 16:33:10.360831 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.360805    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4xr\" (UniqueName: \"kubernetes.io/projected/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-kube-api-access-vf4xr\") pod \"maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.386438 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.386402    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"]
Oct 30 16:33:10.391805 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.391774    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a3cc1e2b-b54bcc94b-9tzwn"]
Oct 30 16:33:10.456116 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.456082    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:10.774067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:10.773965    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"]
Oct 30 16:33:10.779337 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:33:10.779310    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ce2f6d_41de_4fbe_acd9_370f26fc69a3.slice/crio-8940e00490536354145bd3baa7b65e991a09807dd708d72a3a7a8559efddca52 WatchSource:0}: Error finding container 8940e00490536354145bd3baa7b65e991a09807dd708d72a3a7a8559efddca52: Status 404 returned error can't find the container with id 8940e00490536354145bd3baa7b65e991a09807dd708d72a3a7a8559efddca52
Oct 30 16:33:11.342755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:11.342674    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh" event={"ID":"06ce2f6d-41de-4fbe-acd9-370f26fc69a3","Type":"ContainerStarted","Data":"e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02"}
Oct 30 16:33:11.342755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:11.342749    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh" event={"ID":"06ce2f6d-41de-4fbe-acd9-370f26fc69a3","Type":"ContainerStarted","Data":"8940e00490536354145bd3baa7b65e991a09807dd708d72a3a7a8559efddca52"}
Oct 30 16:33:11.343159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:11.342819    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:11.381072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:11.381015    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh" podStartSLOduration=1.082823403 podStartE2EDuration="1.380999405s" totalImagesPullingTime="298.176002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:11.37967177 +0000 UTC m=+621.233550034" watchObservedRunningTime="2026-10-30 16:33:11.380999405 +0000 UTC m=+621.234877671"
Oct 30 16:33:12.299039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:12.299002    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a7e1af-b485-4226-86c3-fa1bc739c14b" path="/var/lib/kubelet/pods/d8a7e1af-b485-4226-86c3-fa1bc739c14b/volumes"
Oct 30 16:33:14.512673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.512576    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"]
Oct 30 16:33:14.518897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.518871    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.531008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.530981    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-420ff3-serving-cert\""
Oct 30 16:33:14.543438 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.543398    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"]
Oct 30 16:33:14.596941 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.596860    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.597463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.597431    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-metrics-certs\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.597647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.597609    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7800f323-b297-49bb-b271-cf1fc2ce4a27-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.597792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.597775    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9cwc\" (UniqueName: \"kubernetes.io/projected/7800f323-b297-49bb-b271-cf1fc2ce4a27-kube-api-access-g9cwc\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.699120 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.699074    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.699347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.699133    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-metrics-certs\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.699347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.699212    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7800f323-b297-49bb-b271-cf1fc2ce4a27-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.699347 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:14.699261    4497 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-serving-cert: secret "maas-api-e2e-worker-w4-420ff3-serving-cert" not found
Oct 30 16:33:14.699347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.699293    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9cwc\" (UniqueName: \"kubernetes.io/projected/7800f323-b297-49bb-b271-cf1fc2ce4a27-kube-api-access-g9cwc\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.699347 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:14.699331    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls podName:7800f323-b297-49bb-b271-cf1fc2ce4a27 nodeName:}" failed. No retries permitted until 2026-10-30 16:33:15.199308672 +0000 UTC m=+625.053186924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls") pod "maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2" (UID: "7800f323-b297-49bb-b271-cf1fc2ce4a27") : secret "maas-api-e2e-worker-w4-420ff3-serving-cert" not found
Oct 30 16:33:14.700249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.700221    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7800f323-b297-49bb-b271-cf1fc2ce4a27-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.702025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.702001    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-metrics-certs\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:14.720765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:14.720736    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9cwc\" (UniqueName: \"kubernetes.io/projected/7800f323-b297-49bb-b271-cf1fc2ce4a27-kube-api-access-g9cwc\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:15.205853 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:15.205808    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:15.208238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:15.208217    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls\") pod \"maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:15.341038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:15.341006    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 16:33:15.431317 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:15.431275    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:15.758775 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:15.758675    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"]
Oct 30 16:33:15.763200 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:33:15.763162    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7800f323_b297_49bb_b271_cf1fc2ce4a27.slice/crio-25f93e0e22f1e13fd7da12f48c811cb6901cdd5fe4f2be5aabce042b11ae4e80 WatchSource:0}: Error finding container 25f93e0e22f1e13fd7da12f48c811cb6901cdd5fe4f2be5aabce042b11ae4e80: Status 404 returned error can't find the container with id 25f93e0e22f1e13fd7da12f48c811cb6901cdd5fe4f2be5aabce042b11ae4e80
Oct 30 16:33:16.369966 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:16.369923    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2" event={"ID":"7800f323-b297-49bb-b271-cf1fc2ce4a27","Type":"ContainerStarted","Data":"6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7"}
Oct 30 16:33:16.370137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:16.369984    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2" event={"ID":"7800f323-b297-49bb-b271-cf1fc2ce4a27","Type":"ContainerStarted","Data":"25f93e0e22f1e13fd7da12f48c811cb6901cdd5fe4f2be5aabce042b11ae4e80"}
Oct 30 16:33:16.370137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:16.370016    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:16.412271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:16.412142    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2" podStartSLOduration=2.034625303 podStartE2EDuration="2.412128473s" totalImagesPullingTime="377.50317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:16.411433638 +0000 UTC m=+626.265311903" watchObservedRunningTime="2026-10-30 16:33:16.412128473 +0000 UTC m=+626.266006738"
Oct 30 16:33:17.353390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:17.353356    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 16:33:18.323519 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:18.323489    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 16:33:18.323674 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:18.323570    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 16:33:19.331125 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:19.331092    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 16:33:21.925993 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:21.925956    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:33:22.406511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:22.406481    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 16:33:24.443572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:24.443541    4497 generic.go:356] "Generic (PLEG): container finished" podID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerID="846072f3c384abf7d12ce69f59a2c2022e06b9953760f5463b561da350194d11" exitCode=0
Oct 30 16:33:24.444014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:24.443625    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" event={"ID":"f019fe5f-39a5-4658-996c-4f91a78231e8","Type":"ContainerDied","Data":"846072f3c384abf7d12ce69f59a2c2022e06b9953760f5463b561da350194d11"}
Oct 30 16:33:24.444074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:24.444055    4497 scope.go:122] "RemoveContainer" containerID="846072f3c384abf7d12ce69f59a2c2022e06b9953760f5463b561da350194d11"
Oct 30 16:33:24.445326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:24.445298    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:24.445382 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:24.445363    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:24.445417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:24.445395    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:25.450922 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:25.450885    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" event={"ID":"f019fe5f-39a5-4658-996c-4f91a78231e8","Type":"ContainerStarted","Data":"81b7dc16b40a246b9037a88936e3709bf76726bf5e61d73f282daa1d6e16fa48"}
Oct 30 16:33:25.976213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:25.975584    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:33:25.996421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:25.996367    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 16:33:25.996614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:25.996445    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 16:33:26.984215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:26.978605    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 16:33:26.984215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:26.978667    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 16:33:27.996425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:27.996386    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 16:33:27.997044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:27.997010    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 16:33:28.977228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:28.973592    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 16:33:28.977228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:28.973655    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 16:33:29.505449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:29.505411    4497 generic.go:356] "Generic (PLEG): container finished" podID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerID="f76a693c91b04fd7917905de51ee79281c802a6013a8b8080b957da62d2445a6" exitCode=0
Oct 30 16:33:29.505846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:29.505476    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" event={"ID":"c5d45ae6-16d3-4605-95ab-19e3b35c4c96","Type":"ContainerDied","Data":"f76a693c91b04fd7917905de51ee79281c802a6013a8b8080b957da62d2445a6"}
Oct 30 16:33:29.506166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:29.506148    4497 scope.go:122] "RemoveContainer" containerID="f76a693c91b04fd7917905de51ee79281c802a6013a8b8080b957da62d2445a6"
Oct 30 16:33:29.507551 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:29.507525    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:29.507608 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:29.507595    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:29.507649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:29.507629    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:33:29.976690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:29.976594    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 16:33:29.976953 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:29.976923    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 16:33:30.240998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:30.240908    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:33:30.516127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:30.516087    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" event={"ID":"c5d45ae6-16d3-4605-95ab-19e3b35c4c96","Type":"ContainerStarted","Data":"263c87bc90091a33dde6d7caf574bc77782d39fa832a39f8fb474ce191e856af"}
Oct 30 16:33:31.312624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:31.312581    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:31.313967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:31.313943    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 16:33:31.314031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:31.313995    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 16:33:31.321035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:31.321007    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:33:31.322460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:31.322437    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 16:33:32.313822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:32.313788    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 16:33:32.314360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:32.313849    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 16:33:33.312630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:33.312593    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 16:33:33.312816 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:33.312647    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 16:33:34.300652 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:34.300618    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:34.312340 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:34.312310    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 16:33:34.312442 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:34.312366    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 16:33:35.312891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:35.312854    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 16:33:35.313466 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:35.312929    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 16:33:36.316357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:36.316326    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:36.317452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:36.317428    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:33:48.953001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:48.952964    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"]
Oct 30 16:33:48.958272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:48.958240    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:48.971484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:48.971455    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"openshift-service-ca.crt\""
Oct 30 16:33:48.980976 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:48.980941    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"kube-root-ca.crt\""
Oct 30 16:33:48.981109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:48.981052    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"default-dockercfg-skqw6\""
Oct 30 16:33:48.981109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:48.981057    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"premium-w3-2d365c-kserve-self-signed-certs\""
Oct 30 16:33:48.983134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:48.983115    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"]
Oct 30 16:33:49.033477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.033438    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4gb\" (UniqueName: \"kubernetes.io/projected/f917b48c-109a-4c9b-8a32-be33e53b320a-kube-api-access-zg4gb\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.033672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.033497    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-tmp-dir\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.033672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.033575    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-home\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.033672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.033623    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-model-cache\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.033672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.033653    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-dshm\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.033897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.033720    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f917b48c-109a-4c9b-8a32-be33e53b320a-tls-certs\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135243    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-model-cache\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135286    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-dshm\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135482 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135311    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f917b48c-109a-4c9b-8a32-be33e53b320a-tls-certs\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135482 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135391    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4gb\" (UniqueName: \"kubernetes.io/projected/f917b48c-109a-4c9b-8a32-be33e53b320a-kube-api-access-zg4gb\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135482 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135432    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-tmp-dir\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135482 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135467    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-home\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135750 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135726    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-model-cache\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135837 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135772    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-home\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.135837 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.135831    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-tmp-dir\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.137764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.137731    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-dshm\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.138240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.138220    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f917b48c-109a-4c9b-8a32-be33e53b320a-tls-certs\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.158085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.158046    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4gb\" (UniqueName: \"kubernetes.io/projected/f917b48c-109a-4c9b-8a32-be33e53b320a-kube-api-access-zg4gb\") pod \"premium-w3-2d365c-kserve-d98ffccc8-7lzst\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") " pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.271181 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.271147    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:49.571260 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.571153    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"]
Oct 30 16:33:49.575530 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:33:49.575495    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf917b48c_109a_4c9b_8a32_be33e53b320a.slice/crio-2d481f991108966ec8a772f396da6a3f46145b57df93cb5098cc85375ad76019 WatchSource:0}: Error finding container 2d481f991108966ec8a772f396da6a3f46145b57df93cb5098cc85375ad76019: Status 404 returned error can't find the container with id 2d481f991108966ec8a772f396da6a3f46145b57df93cb5098cc85375ad76019
Oct 30 16:33:49.616866 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:49.616834    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" event={"ID":"f917b48c-109a-4c9b-8a32-be33e53b320a","Type":"ContainerStarted","Data":"2d481f991108966ec8a772f396da6a3f46145b57df93cb5098cc85375ad76019"}
Oct 30 16:33:50.623244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:50.623146    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" event={"ID":"f917b48c-109a-4c9b-8a32-be33e53b320a","Type":"ContainerStarted","Data":"f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684"}
Oct 30 16:33:51.272356 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:51.272316    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:51.915913 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:33:51.915872    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:33:59.271670 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:59.271628    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:59.284667 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:59.284634    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:33:59.322623 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:59.322567    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podStartSLOduration=11.322553185 podStartE2EDuration="11.322553185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:50.661426543 +0000 UTC m=+660.515304820" watchObservedRunningTime="2026-10-30 16:33:59.322553185 +0000 UTC m=+669.176431450"
Oct 30 16:33:59.684107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:33:59.684030    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:34:18.168842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.168805    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"]
Oct 30 16:34:18.173048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.173026    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.196359 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.196328    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"default-dockercfg-2wjrq\""
Oct 30 16:34:18.196359 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.196341    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"kube-root-ca.crt\""
Oct 30 16:34:18.196532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.196362    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"openshift-service-ca.crt\""
Oct 30 16:34:18.196532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.196382    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"premium-w4-420ff3-kserve-self-signed-certs\""
Oct 30 16:34:18.198864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.198842    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"]
Oct 30 16:34:18.240763 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.240731    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-model-cache\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.240917 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.240778    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-home\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.240917 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.240797    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-dshm\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.241003 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.240939    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9461cfdb-edd6-495d-a013-3a721850d1a2-tls-certs\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.241003 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.240983    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-tmp-dir\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.241117 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.241101    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qt2\" (UniqueName: \"kubernetes.io/projected/9461cfdb-edd6-495d-a013-3a721850d1a2-kube-api-access-d7qt2\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342176    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9461cfdb-edd6-495d-a013-3a721850d1a2-tls-certs\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342236    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-tmp-dir\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342301    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qt2\" (UniqueName: \"kubernetes.io/projected/9461cfdb-edd6-495d-a013-3a721850d1a2-kube-api-access-d7qt2\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342330    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-model-cache\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342365    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-home\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342395    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-dshm\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342715    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-tmp-dir\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342760    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-model-cache\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.342797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.342785    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-home\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.344627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.344603    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-dshm\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.344884 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.344867    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9461cfdb-edd6-495d-a013-3a721850d1a2-tls-certs\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.364448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.364411    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qt2\" (UniqueName: \"kubernetes.io/projected/9461cfdb-edd6-495d-a013-3a721850d1a2-kube-api-access-d7qt2\") pod \"premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") " pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.485442 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.485348    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:18.788963 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:18.788931    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"]
Oct 30 16:34:18.792951 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:34:18.792923    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9461cfdb_edd6_495d_a013_3a721850d1a2.slice/crio-a84f40e47a3b2424c41b215cfda6d81ad33b17308ffb8a18aeac726273591a3f WatchSource:0}: Error finding container a84f40e47a3b2424c41b215cfda6d81ad33b17308ffb8a18aeac726273591a3f: Status 404 returned error can't find the container with id a84f40e47a3b2424c41b215cfda6d81ad33b17308ffb8a18aeac726273591a3f
Oct 30 16:34:19.776423 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:19.776383    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" event={"ID":"9461cfdb-edd6-495d-a013-3a721850d1a2","Type":"ContainerStarted","Data":"b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e"}
Oct 30 16:34:19.776823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:19.776436    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" event={"ID":"9461cfdb-edd6-495d-a013-3a721850d1a2","Type":"ContainerStarted","Data":"a84f40e47a3b2424c41b215cfda6d81ad33b17308ffb8a18aeac726273591a3f"}
Oct 30 16:34:19.813427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:19.813377    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podStartSLOduration=1.8133585939999999 podStartE2EDuration="1.813358594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:19.813312885 +0000 UTC m=+689.667191149" watchObservedRunningTime="2026-10-30 16:34:19.813358594 +0000 UTC m=+689.667236858"
Oct 30 16:34:20.485822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:20.485787    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:21.916836 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:34:21.916801    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:34:28.486016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.485978    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:28.498884 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.498851    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:28.810023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.809985    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"]
Oct 30 16:34:28.814130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.814103    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.826433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.826407    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3503b-openshift-default-dockercfg-xgg9f\""
Oct 30 16:34:28.836012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.835990    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:34:28.842815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.842781    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"]
Oct 30 16:34:28.962692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.962655    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-488hf\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-kube-api-access-488hf\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.962890 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.962712    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.962890 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.962855    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.963014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.962904    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.963014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.963003    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.963130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.963047    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.963130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.963105    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.963263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.963179    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:28.963263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:28.963238    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.063916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.063821    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.063916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.063865    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.063916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.063895    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.063928    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.063946    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.063983    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.064079    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.064104    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.064154    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-488hf\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-kube-api-access-488hf\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064579 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.064399    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064579 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.064473    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.064578    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.064712    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.064866 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.064842    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.066366 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.066346    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.066602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.066582    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.084960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.084926    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.085115 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.085072    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-488hf\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-kube-api-access-488hf\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.128805 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.128771    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:29.447421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.447344    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"]
Oct 30 16:34:29.450936 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:34:29.450906    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3508c50_6126_4b95_9b22_ecf6c6d3636c.slice/crio-69a01a7f69698ff8c0e72adb0ba122416ecea12dd1615ddd5c6d0232a630fc10 WatchSource:0}: Error finding container 69a01a7f69698ff8c0e72adb0ba122416ecea12dd1615ddd5c6d0232a630fc10: Status 404 returned error can't find the container with id 69a01a7f69698ff8c0e72adb0ba122416ecea12dd1615ddd5c6d0232a630fc10
Oct 30 16:34:29.453140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.453108    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:34:29.453228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.453176    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:34:29.453228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.453224    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:34:29.832259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.832223    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" event={"ID":"a3508c50-6126-4b95-9b22-ecf6c6d3636c","Type":"ContainerStarted","Data":"789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638"}
Oct 30 16:34:29.832777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.832286    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" event={"ID":"a3508c50-6126-4b95-9b22-ecf6c6d3636c","Type":"ContainerStarted","Data":"69a01a7f69698ff8c0e72adb0ba122416ecea12dd1615ddd5c6d0232a630fc10"}
Oct 30 16:34:29.899802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:29.899737    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podStartSLOduration=1.899714369 podStartE2EDuration="1.899714369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:29.891329022 +0000 UTC m=+699.745207295" watchObservedRunningTime="2026-10-30 16:34:29.899714369 +0000 UTC m=+699.753592635"
Oct 30 16:34:30.129822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:30.129726    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:30.131292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:30.131269    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 16:34:30.131358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:30.131332    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 16:34:31.130057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:31.130010    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 16:34:31.130579 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:31.130070    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 16:34:32.129901 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:32.129861    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 16:34:32.130086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:32.129923    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 16:34:33.129234 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:33.129165    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 16:34:33.129431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:33.129253    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 16:34:34.129699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:34.129657    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 16:34:34.130138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:34.129728    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 16:34:35.129942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:35.129904    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 16:34:35.130377 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:35.129965    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 16:34:36.135472 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:36.135442    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:36.135912 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:36.135847    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:36.136888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:36.136866    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:34:51.917154 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:34:51.917119    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:34:57.539406 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.539371    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"]
Oct 30 16:34:57.543683 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.543659    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.555853 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.555825    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2d365c-openshift-default-dockercfg-gctmp\""
Oct 30 16:34:57.568210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.568128    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"]
Oct 30 16:34:57.607597 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.607557    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.607774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.607612    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8pt\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-kube-api-access-ln8pt\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.607774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.607720    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.607902 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.607780    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e635afca-a409-4a6f-8f3c-0a74609170fe-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.607902 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.607876    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.607999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.607938    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.607999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.607988    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.608098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.608024    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.608158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.608091    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.708787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.708748    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.708787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.708791    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.708816    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.708845    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.708905    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.708947    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8pt\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-kube-api-access-ln8pt\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.708983    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.709040    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e635afca-a409-4a6f-8f3c-0a74609170fe-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.709064    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.709142    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.709279    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.709488    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709568 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.709543    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.709829 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.709803    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e635afca-a409-4a6f-8f3c-0a74609170fe-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.711573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.711554    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.711652 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.711607    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.729547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.729524    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.729916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.729895    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8pt\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-kube-api-access-ln8pt\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:57.856787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:57.856700    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:58.215272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:58.215140    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"]
Oct 30 16:34:58.220060 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:34:58.220034    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode635afca_a409_4a6f_8f3c_0a74609170fe.slice/crio-41ab70a25eb7622a1b9af9619fb1d949d928e1a1239e75fdbe8b41c5ff1f428e WatchSource:0}: Error finding container 41ab70a25eb7622a1b9af9619fb1d949d928e1a1239e75fdbe8b41c5ff1f428e: Status 404 returned error can't find the container with id 41ab70a25eb7622a1b9af9619fb1d949d928e1a1239e75fdbe8b41c5ff1f428e
Oct 30 16:34:58.222355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:58.222322    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:34:58.222427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:58.222393    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:34:58.222427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:58.222422    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:34:58.995441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:58.995395    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" event={"ID":"e635afca-a409-4a6f-8f3c-0a74609170fe","Type":"ContainerStarted","Data":"b86c7e18af2c1607a2370478d78f5a1e17f3a2cf4cb8b37f6b03e7d16835de47"}
Oct 30 16:34:58.996832 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:58.996797    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" event={"ID":"e635afca-a409-4a6f-8f3c-0a74609170fe","Type":"ContainerStarted","Data":"41ab70a25eb7622a1b9af9619fb1d949d928e1a1239e75fdbe8b41c5ff1f428e"}
Oct 30 16:34:59.049308 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:59.049253    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" podStartSLOduration=2.049238681 podStartE2EDuration="2.049238681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:59.04791915 +0000 UTC m=+728.901797417" watchObservedRunningTime="2026-10-30 16:34:59.049238681 +0000 UTC m=+728.903117023"
Oct 30 16:34:59.857026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:59.856990    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:34:59.858736 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:59.858706    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 16:34:59.858874 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:34:59.858774    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 16:35:00.857176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:00.857140    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 16:35:00.857639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:00.857257    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 16:35:01.857859 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:01.857825    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 16:35:01.858316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:01.857896    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 16:35:02.857847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:02.857813    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 16:35:02.858038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:02.857875    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 16:35:03.857312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:03.857273    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 16:35:03.857506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:03.857331    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 16:35:05.541064 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:05.541031    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:35:05.541897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:05.541876    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:35:05.542936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:05.542917    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:35:16.722223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.722145    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"]
Oct 30 16:35:16.727619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.726794    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.739435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.739408    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 16:35:16.744613 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.744587    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-provider-credentials\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.744758 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.744644    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-plugins-config-volume\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.744823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.744776    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xp6z\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-kube-api-access-8xp6z\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.744903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.744888    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-routing-overlay-volume\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.748849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.748823    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 16:35:16.748998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.748967    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 16:35:16.748998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.748849    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-83b62ed08788\""
Oct 30 16:35:16.749136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.748894    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-996a8bdbc296\""
Oct 30 16:35:16.749136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.748912    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 16:35:16.749136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.748849    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-jb2w5\""
Oct 30 16:35:16.751362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.751342    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"]
Oct 30 16:35:16.845967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.845933    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-routing-overlay-volume\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.846157 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.845975    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-provider-credentials\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.846157 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.846096    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-plugins-config-volume\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.846304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.846205    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp6z\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-kube-api-access-8xp6z\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.846718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.846695    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-routing-overlay-volume\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.846827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.846803    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-plugins-config-volume\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.848620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.848597    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-provider-credentials\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:16.867654 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:16.867614    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xp6z\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-kube-api-access-8xp6z\") pod \"payload-processing-external-model-f95d5b756-h59gl\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") " pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:17.043559 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:17.043526    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:17.366709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:17.366607    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"]
Oct 30 16:35:17.372378 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:35:17.372349    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd7e1db_3f60_4c41_8517_d2a5f69de3fc.slice/crio-9ae69be0cde584136f5a621d1eeec26b2c5aec9fc56ae5890060d99eea2cc434 WatchSource:0}: Error finding container 9ae69be0cde584136f5a621d1eeec26b2c5aec9fc56ae5890060d99eea2cc434: Status 404 returned error can't find the container with id 9ae69be0cde584136f5a621d1eeec26b2c5aec9fc56ae5890060d99eea2cc434
Oct 30 16:35:18.113546 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:18.113501    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl" event={"ID":"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc","Type":"ContainerStarted","Data":"9aa2017b2b56b52fa041f8c261b40e21327d5a8d9f6d826edc7c1e3a868233d2"}
Oct 30 16:35:18.114101 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:18.113577    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl" event={"ID":"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc","Type":"ContainerStarted","Data":"9ae69be0cde584136f5a621d1eeec26b2c5aec9fc56ae5890060d99eea2cc434"}
Oct 30 16:35:18.114335 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:18.114307    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:18.153433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:18.153357    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl" podStartSLOduration=2.153312957 podStartE2EDuration="2.153312957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:18.151617254 +0000 UTC m=+748.005495518" watchObservedRunningTime="2026-10-30 16:35:18.153312957 +0000 UTC m=+748.007191223"
Oct 30 16:35:21.923231 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:35:21.923155    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:35:27.005794 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.005750    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"]
Oct 30 16:35:27.009975 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.009958    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.034070 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.034039    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"]
Oct 30 16:35:27.071550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.071430    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.071550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.071516    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37dd2c72-4998-4903-a89b-6c75511aaf65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.072388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.071877    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.072388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.071939    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.072388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.072044    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.072388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.072073    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crbjc\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-kube-api-access-crbjc\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.072388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.072110    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.072388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.072225    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.072388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.072257    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173060    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173110    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37dd2c72-4998-4903-a89b-6c75511aaf65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173152    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173177    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173290    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173320    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crbjc\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-kube-api-access-crbjc\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173358    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173462    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.173524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173494    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.174012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.173969    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.174574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.174397    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.174574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.174452    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37dd2c72-4998-4903-a89b-6c75511aaf65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.174574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.174526    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.174574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.174542    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.176328 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.176302    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.176647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.176627    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.195018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.194984    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.195155 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.195074    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crbjc\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-kube-api-access-crbjc\") pod \"maas-default-gateway-openshift-default-7854949c9-fr5gj\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.325290 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.325164    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:27.649932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.649909    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"]
Oct 30 16:35:27.652619 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:35:27.652589    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37dd2c72_4998_4903_a89b_6c75511aaf65.slice/crio-cd5f899ef6d4b9fca9dc2dc0e76b1d90c898597ddb566d5d8cd05213501eb944 WatchSource:0}: Error finding container cd5f899ef6d4b9fca9dc2dc0e76b1d90c898597ddb566d5d8cd05213501eb944: Status 404 returned error can't find the container with id cd5f899ef6d4b9fca9dc2dc0e76b1d90c898597ddb566d5d8cd05213501eb944
Oct 30 16:35:27.655012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.654976    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:27.655100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.655054    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:27.655100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:27.655090    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:28.204476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:28.204435    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" event={"ID":"37dd2c72-4998-4903-a89b-6c75511aaf65","Type":"ContainerStarted","Data":"c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700"}
Oct 30 16:35:28.204964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:28.204500    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" event={"ID":"37dd2c72-4998-4903-a89b-6c75511aaf65","Type":"ContainerStarted","Data":"cd5f899ef6d4b9fca9dc2dc0e76b1d90c898597ddb566d5d8cd05213501eb944"}
Oct 30 16:35:28.264579 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:28.264516    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podStartSLOduration=2.264500959 podStartE2EDuration="2.264500959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:28.263845606 +0000 UTC m=+758.117723871" watchObservedRunningTime="2026-10-30 16:35:28.264500959 +0000 UTC m=+758.118379224"
Oct 30 16:35:28.325609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:28.325580    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:28.326971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:28.326945    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 16:35:28.327112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:28.327008    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 16:35:29.128978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:29.128945    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:29.325624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:29.325581    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 16:35:29.326036 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:29.325645    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 16:35:29.945450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:29.945410    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"]
Oct 30 16:35:30.325821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:30.325787    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 16:35:30.326245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:30.325834    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 16:35:31.326341 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:31.326311    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 16:35:31.326920 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:31.326361    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 16:35:32.326507 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:32.326424    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 16:35:32.326507 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:32.326494    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 16:35:33.326095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:33.326053    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 16:35:33.326308 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:33.326116    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 16:35:34.326207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:34.326140    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 16:35:34.326834 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:34.326799    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 16:35:34.946783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:34.946714    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl" podUID="2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" containerName="payload-processing" containerID="cri-o://9aa2017b2b56b52fa041f8c261b40e21327d5a8d9f6d826edc7c1e3a868233d2" gracePeriod=25
Oct 30 16:35:35.253277 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.253241    4497 generic.go:356] "Generic (PLEG): container finished" podID="2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" containerID="9aa2017b2b56b52fa041f8c261b40e21327d5a8d9f6d826edc7c1e3a868233d2" exitCode=0
Oct 30 16:35:35.253433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.253373    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl" event={"ID":"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc","Type":"ContainerDied","Data":"9aa2017b2b56b52fa041f8c261b40e21327d5a8d9f6d826edc7c1e3a868233d2"}
Oct 30 16:35:35.340104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.339656    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 16:35:35.340104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.339717    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 16:35:35.363623 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.363600    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:35.482735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.482684    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-routing-overlay-volume\") pod \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") "
Oct 30 16:35:35.483153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.483130    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-provider-credentials\") pod \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") "
Oct 30 16:35:35.483329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.483311    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-plugins-config-volume\") pod \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") "
Oct 30 16:35:35.484534 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.484506    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-plugins-config-volume" pod "2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" (UID: "2fd7e1db-3f60-4c41-8517-d2a5f69de3fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:35:35.484886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.484858    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-kube-api-access-8xp6z\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-kube-api-access-8xp6z\") pod \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\" (UID: \"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc\") "
Oct 30 16:35:35.485938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.485916    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-routing-overlay-volume" pod "2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" (UID: "2fd7e1db-3f60-4c41-8517-d2a5f69de3fc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:35:35.489095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.486511    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-provider-credentials" pod "2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" (UID: "2fd7e1db-3f60-4c41-8517-d2a5f69de3fc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:35:35.489095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.488679    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-kube-api-access-8xp6z" pod "2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" (UID: "2fd7e1db-3f60-4c41-8517-d2a5f69de3fc"). InnerVolumeSpecName "kube-api-access-8xp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:35:35.489095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.488986    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:35:35.592041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.591992    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xp6z\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-kube-api-access-8xp6z\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:35:35.592041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.592039    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:35:35.592356 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:35.592055    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:35:36.259993 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.259958    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl" event={"ID":"2fd7e1db-3f60-4c41-8517-d2a5f69de3fc","Type":"ContainerDied","Data":"9ae69be0cde584136f5a621d1eeec26b2c5aec9fc56ae5890060d99eea2cc434"}
Oct 30 16:35:36.259993 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.259985    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"
Oct 30 16:35:36.260302 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.260022    4497 scope.go:122] "RemoveContainer" containerID="9aa2017b2b56b52fa041f8c261b40e21327d5a8d9f6d826edc7c1e3a868233d2"
Oct 30 16:35:36.304557 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.304526    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"]
Oct 30 16:35:36.313904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.313876    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f95d5b756-h59gl"]
Oct 30 16:35:36.523797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.523651    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"]
Oct 30 16:35:36.525044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.525014    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" containerName="payload-processing"
Oct 30 16:35:36.525044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.525041    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" containerName="payload-processing"
Oct 30 16:35:36.525276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.525121    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" containerName="payload-processing"
Oct 30 16:35:36.529781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.529749    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.542227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.542171    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"openshift-service-ca.crt\""
Oct 30 16:35:36.542375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.542171    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"kube-root-ca.crt\""
Oct 30 16:35:36.542566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.542547    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"e2e-conflict-a-239c91ff-kserve-self-signed-certs\""
Oct 30 16:35:36.542651 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.542615    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"default-dockercfg-rzlfm\""
Oct 30 16:35:36.554224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.549412    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"]
Oct 30 16:35:36.603820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.603777    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-home\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.604005 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.603923    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-model-cache\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.604005 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.603984    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8km\" (UniqueName: \"kubernetes.io/projected/0fbf56dd-e248-43b3-b429-27705650b1a7-kube-api-access-lq8km\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.604133 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.604121    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbf56dd-e248-43b3-b429-27705650b1a7-tls-certs\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.604236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.604212    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-tmp-dir\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.604298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.604269    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-dshm\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.705274 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705239    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbf56dd-e248-43b3-b429-27705650b1a7-tls-certs\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.705489 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705290    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-tmp-dir\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.705489 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705324    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-dshm\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.705489 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705347    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-home\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.705489 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705429    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-model-cache\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.705695 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705493    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8km\" (UniqueName: \"kubernetes.io/projected/0fbf56dd-e248-43b3-b429-27705650b1a7-kube-api-access-lq8km\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.705946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705891    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-home\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.705946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705916    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-model-cache\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.706149 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.705987    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-tmp-dir\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.707743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.707705    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-dshm\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.707998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.707977    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbf56dd-e248-43b3-b429-27705650b1a7-tls-certs\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.727012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.726980    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8km\" (UniqueName: \"kubernetes.io/projected/0fbf56dd-e248-43b3-b429-27705650b1a7-kube-api-access-lq8km\") pod \"e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:36.822081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.821989    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:36.822632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.822548    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:36.823424 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.823405    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:35:36.846662 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:36.846628    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:37.166005 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.165886    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"]
Oct 30 16:35:37.267265 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.267215    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" event={"ID":"0fbf56dd-e248-43b3-b429-27705650b1a7","Type":"ContainerStarted","Data":"a094ac2e017075b5942bf962c2e72518400335f3778bf357f50efca47377b522"}
Oct 30 16:35:37.457616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.457521    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"]
Oct 30 16:35:37.462217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.462182    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.475234 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.475206    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-503ee2727cb9\""
Oct 30 16:35:37.475387 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.475360    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-996a8bdbc296\""
Oct 30 16:35:37.475387 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.475366    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-83b62ed08788\""
Oct 30 16:35:37.475484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.475387    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-jb2w5\""
Oct 30 16:35:37.486470 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.486443    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"]
Oct 30 16:35:37.519950 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.519913    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-provider-credentials\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.520150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.519966    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-plugins-config-volume\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.520150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.520101    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-routing-overlay-volume\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.520150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.520143    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt69w\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-kube-api-access-vt69w\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.555747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.555715    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 16:35:37.555747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.555720    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 16:35:37.556173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.555752    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 16:35:37.556173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.555769    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 16:35:37.621022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.620990    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-routing-overlay-volume\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.621246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.621047    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt69w\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-kube-api-access-vt69w\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.621246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.621124    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-provider-credentials\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.621246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.621157    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-plugins-config-volume\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.621812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.621787    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-routing-overlay-volume\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.621926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.621907    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-plugins-config-volume\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.623740 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.623718    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-provider-credentials\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.642532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.642500    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt69w\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-kube-api-access-vt69w\") pod \"payload-processing-external-model-779c475d6f-ntbng\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") " pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:37.775776 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:37.775740    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:38.079720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.079631    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"]
Oct 30 16:35:38.083665 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:35:38.083633    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf548aa34_529f_49e7_85fa_f035df3a5255.slice/crio-cb93816537fedf570d4aa2a66573cc6d7b29e5c00cb8ce0031a3c18508f15896 WatchSource:0}: Error finding container cb93816537fedf570d4aa2a66573cc6d7b29e5c00cb8ce0031a3c18508f15896: Status 404 returned error can't find the container with id cb93816537fedf570d4aa2a66573cc6d7b29e5c00cb8ce0031a3c18508f15896
Oct 30 16:35:38.277347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.277292    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" event={"ID":"0fbf56dd-e248-43b3-b429-27705650b1a7","Type":"ContainerStarted","Data":"0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511"}
Oct 30 16:35:38.279240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.279181    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng" event={"ID":"f548aa34-529f-49e7-85fa-f035df3a5255","Type":"ContainerStarted","Data":"508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab"}
Oct 30 16:35:38.279409 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.279258    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng" event={"ID":"f548aa34-529f-49e7-85fa-f035df3a5255","Type":"ContainerStarted","Data":"cb93816537fedf570d4aa2a66573cc6d7b29e5c00cb8ce0031a3c18508f15896"}
Oct 30 16:35:38.279409 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.279387    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:38.301361 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.301321    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd7e1db-3f60-4c41-8517-d2a5f69de3fc" path="/var/lib/kubelet/pods/2fd7e1db-3f60-4c41-8517-d2a5f69de3fc/volumes"
Oct 30 16:35:38.315118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.315054    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" podStartSLOduration=2.315034089 podStartE2EDuration="2.315034089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:38.313603627 +0000 UTC m=+768.167481897" watchObservedRunningTime="2026-10-30 16:35:38.315034089 +0000 UTC m=+768.168912355"
Oct 30 16:35:38.355747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.355631    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng" podStartSLOduration=1.355606629 podStartE2EDuration="1.355606629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:38.353480635 +0000 UTC m=+768.207358899" watchObservedRunningTime="2026-10-30 16:35:38.355606629 +0000 UTC m=+768.209484895"
Oct 30 16:35:38.847364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:38.847334    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:42.076268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.076230    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"]
Oct 30 16:35:42.080717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.080692    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.110234 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.109664    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"]
Oct 30 16:35:42.174317 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174285    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.174475 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174323    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.174475 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174342    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.174475 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174363    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vbx\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-kube-api-access-k6vbx\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.174578 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174514    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.174578 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174543    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.174653 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174634    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.174702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174691    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b58f985-587c-4614-9dde-96394ccf0cc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.174792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.174780    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.275975 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.275938    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.275975 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.275980    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276001    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276023    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276135    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vbx\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-kube-api-access-k6vbx\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276328    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276356    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276379    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276412    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276618 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276499    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b58f985-587c-4614-9dde-96394ccf0cc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276618 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276556    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276741 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276720    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.276808 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.276783    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.277171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.277154    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b58f985-587c-4614-9dde-96394ccf0cc2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.278439 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.278423    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.278739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.278724    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.297581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.297551    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.297731 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.297708    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vbx\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-kube-api-access-k6vbx\") pod \"maas-default-gateway-openshift-default-7854949c9-6v4bs\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.394449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.394361    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:42.729117 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.729035    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"]
Oct 30 16:35:42.732594 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:35:42.732565    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b58f985_587c_4614_9dde_96394ccf0cc2.slice/crio-7dbd58b010d5b8b8aea16bc8c0fc92a2f1d254ff61f7345d1d46595ce0df7237 WatchSource:0}: Error finding container 7dbd58b010d5b8b8aea16bc8c0fc92a2f1d254ff61f7345d1d46595ce0df7237: Status 404 returned error can't find the container with id 7dbd58b010d5b8b8aea16bc8c0fc92a2f1d254ff61f7345d1d46595ce0df7237
Oct 30 16:35:42.738778 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.738745    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:42.738865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.738814    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:42.738865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:42.738841    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:43.330216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:43.328418    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" event={"ID":"0b58f985-587c-4614-9dde-96394ccf0cc2","Type":"ContainerStarted","Data":"80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9"}
Oct 30 16:35:43.330216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:43.328489    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" event={"ID":"0b58f985-587c-4614-9dde-96394ccf0cc2","Type":"ContainerStarted","Data":"7dbd58b010d5b8b8aea16bc8c0fc92a2f1d254ff61f7345d1d46595ce0df7237"}
Oct 30 16:35:43.394209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:43.393792    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podStartSLOduration=1.393758737 podStartE2EDuration="1.393758737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:43.380675224 +0000 UTC m=+773.234553491" watchObservedRunningTime="2026-10-30 16:35:43.393758737 +0000 UTC m=+773.247637018"
Oct 30 16:35:43.412468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:43.403475    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:43.423668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:43.415326    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 16:35:43.423668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:43.415397    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 16:35:44.401211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:44.400677    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 16:35:44.401211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:44.400744    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 16:35:45.407211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:45.403585    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 16:35:45.407211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:45.403649    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 16:35:46.396063 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:46.395586    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 16:35:46.396063 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:46.395651    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 16:35:46.846821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:46.846787    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:46.860407 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:46.860370    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:47.396642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:47.396602    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 16:35:47.396825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:47.396661    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 16:35:47.455278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:47.455248    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:35:48.396732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:48.396681    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 16:35:48.397133 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:48.396761    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 16:35:49.288716 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:49.288688    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:35:49.394926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:49.394892    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 16:35:49.395090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:49.394944    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 16:35:50.395140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:50.395106    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 16:35:50.395563 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:50.395157    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 16:35:51.693860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:51.693825    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:51.694272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:51.694220    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:51.695033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:51.695014    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:35:51.918352 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:35:51.918318    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:35:58.206217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.206165    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"]
Oct 30 16:35:58.211628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.211610    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.234117 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.234091    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"]
Oct 30 16:35:58.317278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317244    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.317278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317279    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.317508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317305    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.317508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317439    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwl2\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-kube-api-access-jkwl2\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.317508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317486    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.317601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317555    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.317601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317596    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.317669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317650    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.317774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.317749    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.418933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.418894    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.418956    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.418979    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.418998    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419041    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwl2\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-kube-api-access-jkwl2\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419102    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419153    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419230    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419308    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419554 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419490    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419613 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419575    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419771 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419740    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419901 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419880    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.419979 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.419960    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.421494 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.421469    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.421651 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.421633    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.440101 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.440066    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.440271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.440181    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwl2\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-kube-api-access-jkwl2\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.524790 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.524756    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:35:58.855944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.855863    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"]
Oct 30 16:35:58.859602 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:35:58.859572    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbff0ae7_4fa4_45d9_ad1d_fd8ec7b31690.slice/crio-de1bb7d93e20e92bfab2ad6ed2c68e2f17fba0c86926d5c5ec0e249f3fad260d WatchSource:0}: Error finding container de1bb7d93e20e92bfab2ad6ed2c68e2f17fba0c86926d5c5ec0e249f3fad260d: Status 404 returned error can't find the container with id de1bb7d93e20e92bfab2ad6ed2c68e2f17fba0c86926d5c5ec0e249f3fad260d
Oct 30 16:35:58.861991 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.861954    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:58.862067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.862026    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:58.862111 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:58.862068    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:35:59.597221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:59.594579    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" event={"ID":"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690","Type":"ContainerStarted","Data":"0dbba1838c49847dfa69ea85f5bbc2c3b5421bdbd2a6e4c00364627db05f7833"}
Oct 30 16:35:59.597221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:59.594650    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" event={"ID":"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690","Type":"ContainerStarted","Data":"de1bb7d93e20e92bfab2ad6ed2c68e2f17fba0c86926d5c5ec0e249f3fad260d"}
Oct 30 16:35:59.646838 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:35:59.646784    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podStartSLOduration=1.646765357 podStartE2EDuration="1.646765357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:59.645097044 +0000 UTC m=+789.498975310" watchObservedRunningTime="2026-10-30 16:35:59.646765357 +0000 UTC m=+789.500643623"
Oct 30 16:36:00.525548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:00.525515    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:36:00.526949 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:00.526917    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 16:36:00.527058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:00.526988    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 16:36:01.525960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:01.525920    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 16:36:01.526399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:01.525980    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 16:36:02.529232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:02.528610    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 16:36:02.529232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:02.528687    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 16:36:03.294239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:03.268517    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"]
Oct 30 16:36:03.537016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:03.536980    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 16:36:03.537655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:03.537620    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 16:36:04.623214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:04.618339    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 16:36:04.623214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:04.618421    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 16:36:04.623214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:04.618858    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"]
Oct 30 16:36:04.642302 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:04.628686    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:04.678237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:04.643538    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"distinct-w3-2d365c-kserve-self-signed-certs\""
Oct 30 16:36:04.715132 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:04.715005    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"]
Oct 30 16:36:05.017415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.016905    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-model-cache\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.017415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.016960    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-tmp-dir\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.017415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.017075    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwcs\" (UniqueName: \"kubernetes.io/projected/1a890f36-432f-4f13-b9c4-067c99f91990-kube-api-access-gxwcs\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.017415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.017108    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-home\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.017415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.017215    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-dshm\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.017415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.017244    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a890f36-432f-4f13-b9c4-067c99f91990-tls-certs\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.122210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.122097    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-dshm\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.122210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.122173    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a890f36-432f-4f13-b9c4-067c99f91990-tls-certs\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.122494 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.122309    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-model-cache\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.122494 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.122340    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-tmp-dir\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.122628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.122495    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwcs\" (UniqueName: \"kubernetes.io/projected/1a890f36-432f-4f13-b9c4-067c99f91990-kube-api-access-gxwcs\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.122628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.122544    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-home\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.127224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.123244    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-home\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.127224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.123977    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-tmp-dir\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.127224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.124972    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-model-cache\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.146052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.146017    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a890f36-432f-4f13-b9c4-067c99f91990-tls-certs\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.154211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.150211    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-dshm\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.157784 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.157742    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwcs\" (UniqueName: \"kubernetes.io/projected/1a890f36-432f-4f13-b9c4-067c99f91990-kube-api-access-gxwcs\") pod \"distinct-w3-2d365c-kserve-57f8f765-2fd2j\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.257502 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.257469    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:05.525815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.525704    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 16:36:05.526105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:05.526056    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 16:36:06.324216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:06.298615    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"]
Oct 30 16:36:06.324216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:06.304918    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-pkh5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:36:06.324216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:06.305010    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:36:06.526410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:06.525933    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 16:36:06.526410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:06.526012    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 16:36:07.261248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:07.261203    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" event={"ID":"1a890f36-432f-4f13-b9c4-067c99f91990","Type":"ContainerStarted","Data":"70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432"}
Oct 30 16:36:07.261433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:07.261263    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" event={"ID":"1a890f36-432f-4f13-b9c4-067c99f91990","Type":"ContainerStarted","Data":"e3b199d8cd666ea1e3d1a8f058b33bf1677ef6259516862b6544eff76265292f"}
Oct 30 16:36:07.302947 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:07.302881    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podStartSLOduration=3.302861262 podStartE2EDuration="3.302861262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:07.299671081 +0000 UTC m=+797.153549347" watchObservedRunningTime="2026-10-30 16:36:07.302861262 +0000 UTC m=+797.156739529"
Oct 30 16:36:07.529516 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:07.529434    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:36:07.529998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:07.529733    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:36:07.530556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:07.530539    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 16:36:08.258942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.258910    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:08.421303 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.421221    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng" podUID="f548aa34-529f-49e7-85fa-f035df3a5255" containerName="payload-processing" containerID="cri-o://508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab" gracePeriod=25
Oct 30 16:36:08.703968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.703944    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:36:08.839043 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.838962    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-provider-credentials\") pod \"f548aa34-529f-49e7-85fa-f035df3a5255\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") "
Oct 30 16:36:08.839219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.839079    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-routing-overlay-volume\") pod \"f548aa34-529f-49e7-85fa-f035df3a5255\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") "
Oct 30 16:36:08.839219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.839109    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-plugins-config-volume\") pod \"f548aa34-529f-49e7-85fa-f035df3a5255\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") "
Oct 30 16:36:08.839219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.839144    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-kube-api-access-vt69w\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-kube-api-access-vt69w\") pod \"f548aa34-529f-49e7-85fa-f035df3a5255\" (UID: \"f548aa34-529f-49e7-85fa-f035df3a5255\") "
Oct 30 16:36:08.839525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.839494    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-plugins-config-volume" pod "f548aa34-529f-49e7-85fa-f035df3a5255" (UID: "f548aa34-529f-49e7-85fa-f035df3a5255"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:36:08.839525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.839503    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-routing-overlay-volume" pod "f548aa34-529f-49e7-85fa-f035df3a5255" (UID: "f548aa34-529f-49e7-85fa-f035df3a5255"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:36:08.839684 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.839630    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:36:08.839684 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.839645    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f548aa34-529f-49e7-85fa-f035df3a5255-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:36:08.841153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.841128    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-provider-credentials" pod "f548aa34-529f-49e7-85fa-f035df3a5255" (UID: "f548aa34-529f-49e7-85fa-f035df3a5255"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:36:08.841278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.841207    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-kube-api-access-vt69w" pod "f548aa34-529f-49e7-85fa-f035df3a5255" (UID: "f548aa34-529f-49e7-85fa-f035df3a5255"). InnerVolumeSpecName "kube-api-access-vt69w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:36:08.940568 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.940535    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt69w\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-kube-api-access-vt69w\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:36:08.940568 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:08.940564    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f548aa34-529f-49e7-85fa-f035df3a5255-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:36:09.274550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.274513    4497 generic.go:356] "Generic (PLEG): container finished" podID="f548aa34-529f-49e7-85fa-f035df3a5255" containerID="508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab" exitCode=0
Oct 30 16:36:09.274733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.274560    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"
Oct 30 16:36:09.274733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.274592    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng" event={"ID":"f548aa34-529f-49e7-85fa-f035df3a5255","Type":"ContainerDied","Data":"508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab"}
Oct 30 16:36:09.274733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.274657    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-779c475d6f-ntbng" event={"ID":"f548aa34-529f-49e7-85fa-f035df3a5255","Type":"ContainerDied","Data":"cb93816537fedf570d4aa2a66573cc6d7b29e5c00cb8ce0031a3c18508f15896"}
Oct 30 16:36:09.274733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.274688    4497 scope.go:122] "RemoveContainer" containerID="508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab"
Oct 30 16:36:09.287640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.287623    4497 scope.go:122] "RemoveContainer" containerID="508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab"
Oct 30 16:36:09.287901 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:36:09.287879    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab\": container with ID starting with 508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab not found: ID does not exist" containerID="508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab"
Oct 30 16:36:09.287966 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.287908    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab"} err="failed to get container status \"508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab\": rpc error: code = NotFound desc = could not find container \"508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab\": container with ID starting with 508efecdfd230e99ec6e320feba99704df7fb00f01ca66da23a86e95362f10ab not found: ID does not exist"
Oct 30 16:36:09.321745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.321716    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"]
Oct 30 16:36:09.330967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:09.330927    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-779c475d6f-ntbng"]
Oct 30 16:36:10.301029 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:10.300992    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f548aa34-529f-49e7-85fa-f035df3a5255" path="/var/lib/kubelet/pods/f548aa34-529f-49e7-85fa-f035df3a5255/volumes"
Oct 30 16:36:15.261920 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.261881    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:15.296575 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.296546    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:15.350263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.350230    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:36:15.925344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.925303    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"]
Oct 30 16:36:15.926078 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.926052    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f548aa34-529f-49e7-85fa-f035df3a5255" containerName="payload-processing"
Oct 30 16:36:15.926078 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.926075    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f548aa34-529f-49e7-85fa-f035df3a5255" containerName="payload-processing"
Oct 30 16:36:15.926305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.926288    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f548aa34-529f-49e7-85fa-f035df3a5255" containerName="payload-processing"
Oct 30 16:36:15.931724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.931698    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:15.960770 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.960684    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"e2e-conflict-b-239c91ff-kserve-self-signed-certs\""
Oct 30 16:36:15.966782 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.966750    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"]
Oct 30 16:36:15.980030 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.979996    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-dshm\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:15.980356 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.980331    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-model-cache\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:15.980569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.980551    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tmp-dir\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:15.980754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.980734    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htn5\" (UniqueName: \"kubernetes.io/projected/8a7d8da4-e26f-4ed1-852d-6f768a50450d-kube-api-access-6htn5\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:15.980916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.980899    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tls-certs\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:15.981181 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:15.981163    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-home\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.082253 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.082178    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-home\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.082476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.082331    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-dshm\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.082476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.082368    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-model-cache\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.082476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.082409    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tmp-dir\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.082476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.082439    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htn5\" (UniqueName: \"kubernetes.io/projected/8a7d8da4-e26f-4ed1-852d-6f768a50450d-kube-api-access-6htn5\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.082476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.082473    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tls-certs\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.082775 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.082709    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-home\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.083027 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.082999    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-model-cache\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.083695 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.083669    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tmp-dir\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.085969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.085922    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tls-certs\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.086138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.086114    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-dshm\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.106497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.106463    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htn5\" (UniqueName: \"kubernetes.io/projected/8a7d8da4-e26f-4ed1-852d-6f768a50450d-kube-api-access-6htn5\") pod \"e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") " pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.261128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.261078    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:16.698171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:16.698130    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"]
Oct 30 16:36:17.385293 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:17.385224    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf" event={"ID":"8a7d8da4-e26f-4ed1-852d-6f768a50450d","Type":"ContainerStarted","Data":"063c561fcca25616e2e1e06c2b931c634135aa8da2ab24a8a2dd394c87675f6a"}
Oct 30 16:36:18.430210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:18.430017    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf" event={"ID":"8a7d8da4-e26f-4ed1-852d-6f768a50450d","Type":"ContainerStarted","Data":"6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3"}
Oct 30 16:36:18.505955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:18.501410    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf" podStartSLOduration=3.501391804 podStartE2EDuration="3.501391804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:18.500841665 +0000 UTC m=+808.354719929" watchObservedRunningTime="2026-10-30 16:36:18.501391804 +0000 UTC m=+808.355270069"
Oct 30 16:36:19.261952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:19.261916    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:21.916240 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:36:21.916204    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:36:26.261682 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:26.261639    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:26.274392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:26.274363    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:26.493992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:26.493945    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:36:29.164012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.163972    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"]
Oct 30 16:36:29.168708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.168680    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179462    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179667 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179611    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3099ed01-f441-4de2-bb5b-cbc6d167c809-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179667    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179704    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179794    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179831    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179869    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179888    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcflp\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-kube-api-access-rcflp\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.179977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.179952    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.195520 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.195484    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"]
Oct 30 16:36:29.280848 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.280808    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3099ed01-f441-4de2-bb5b-cbc6d167c809-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.280858    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.280887    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.280911    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.280948    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281001    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281027    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcflp\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-kube-api-access-rcflp\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281052    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281104    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281354    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281589 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281556    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281676    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281760 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281699    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3099ed01-f441-4de2-bb5b-cbc6d167c809-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.281839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.281795    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.283606 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.283577    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.283719 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.283688    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.301990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.301959    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.302172 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.302132    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcflp\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-kube-api-access-rcflp\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.482311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.482224    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:29.804148 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.804108    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"]
Oct 30 16:36:29.808795 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:36:29.808765    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3099ed01_f441_4de2_bb5b_cbc6d167c809.slice/crio-2c43fc60a784594e986481fa8a6363863eb2011ca75c2e543cd3b730ea693b78 WatchSource:0}: Error finding container 2c43fc60a784594e986481fa8a6363863eb2011ca75c2e543cd3b730ea693b78: Status 404 returned error can't find the container with id 2c43fc60a784594e986481fa8a6363863eb2011ca75c2e543cd3b730ea693b78
Oct 30 16:36:29.811332 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.811298    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:36:29.811420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.811371    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:36:29.811420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:29.811403    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:36:30.508732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:30.508693    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" event={"ID":"3099ed01-f441-4de2-bb5b-cbc6d167c809","Type":"ContainerStarted","Data":"94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323"}
Oct 30 16:36:30.509131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:30.508752    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" event={"ID":"3099ed01-f441-4de2-bb5b-cbc6d167c809","Type":"ContainerStarted","Data":"2c43fc60a784594e986481fa8a6363863eb2011ca75c2e543cd3b730ea693b78"}
Oct 30 16:36:30.568676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:30.568616    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podStartSLOduration=1.5686011020000001 podStartE2EDuration="1.568601102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:30.55788024 +0000 UTC m=+820.411758506" watchObservedRunningTime="2026-10-30 16:36:30.568601102 +0000 UTC m=+820.422479368"
Oct 30 16:36:31.489427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:31.487602    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:31.490100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:31.489727    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 16:36:31.490100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:31.489796    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 16:36:32.496121 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:32.496083    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 16:36:32.496619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:32.496411    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 16:36:33.488429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:33.488391    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 16:36:33.488849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:33.488820    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 16:36:34.486213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:34.485742    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 16:36:34.486213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:34.485825    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 16:36:35.484595 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:35.484556    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 16:36:35.484897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:35.484867    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 16:36:36.483516 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:36.483479    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 16:36:36.483959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:36.483545    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 16:36:38.178810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:38.178747    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:38.179307 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:38.179078    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:38.181436 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:38.181417    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:36:43.377111 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.377073    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"]
Oct 30 16:36:43.384558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.384525    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.427928 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.427897    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"]
Oct 30 16:36:43.498824 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.498788    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.498999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.498832    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.498999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.498858    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.498999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.498916    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-kube-api-access-smkr2\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.499118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.499029    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.499118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.499080    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.499199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.499149    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.499199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.499172    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.499312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.499298    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600494 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600455    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600519    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600557    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600584    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600663    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600802    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600833    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600859    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600883    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-kube-api-access-smkr2\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600917    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.600978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.600942    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.601286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.601265    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.601333 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.601310    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.601416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.601398    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.603032 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.603004    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.603215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.603182    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.622659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.622617    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-kube-api-access-smkr2\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.622819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.622775    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:43.722351 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:43.722265    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:44.049604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:44.049567    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"]
Oct 30 16:36:44.053237 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:36:44.053208    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edcb04e_c6ad_4b52_9b15_8ad571f7c7b7.slice/crio-9c84b5df3f67189f3203ddd203c8214d539af936fda6713b05cf4a2abebcbbb6 WatchSource:0}: Error finding container 9c84b5df3f67189f3203ddd203c8214d539af936fda6713b05cf4a2abebcbbb6: Status 404 returned error can't find the container with id 9c84b5df3f67189f3203ddd203c8214d539af936fda6713b05cf4a2abebcbbb6
Oct 30 16:36:44.055600 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:44.055566    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:36:44.055676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:44.055639    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:36:44.055676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:44.055670    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:36:44.593707 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:44.593670    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" event={"ID":"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7","Type":"ContainerStarted","Data":"4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634"}
Oct 30 16:36:44.594116 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:44.593729    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" event={"ID":"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7","Type":"ContainerStarted","Data":"9c84b5df3f67189f3203ddd203c8214d539af936fda6713b05cf4a2abebcbbb6"}
Oct 30 16:36:44.647453 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:44.647391    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podStartSLOduration=1.647366428 podStartE2EDuration="1.647366428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:44.644765135 +0000 UTC m=+834.498643400" watchObservedRunningTime="2026-10-30 16:36:44.647366428 +0000 UTC m=+834.501244694"
Oct 30 16:36:44.725265 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:44.725145    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:45.724351 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:45.724313    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 16:36:45.724844 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:45.724373    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 16:36:46.722679 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:46.722639    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 16:36:46.722896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:46.722692    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 16:36:47.722722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:47.722664    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 16:36:47.723259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:47.722728    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 16:36:48.759213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:48.737104    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 16:36:48.759213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:48.737222    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 16:36:49.729213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:49.722791    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 16:36:49.729213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:49.722851    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 16:36:50.328213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:50.326508    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:36:50.450953 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:50.450907    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 16:36:50.723749 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:50.723642    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 16:36:50.724066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:50.724006    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 16:36:51.725245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:51.723078    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 16:36:51.725245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:51.723140    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 16:36:51.917530 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:36:51.917489    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:36:52.727715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:52.727677    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:52.728166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:52.728094    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:52.728796 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:52.728779    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:36:56.303554 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:36:56.303517    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"]
Oct 30 16:37:03.980798 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:03.977713    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"]
Oct 30 16:37:11.331625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.331569    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf" podUID="8a7d8da4-e26f-4ed1-852d-6f768a50450d" containerName="main" containerID="cri-o://6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3" gracePeriod=45
Oct 30 16:37:11.483065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.482790    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf" podUID="8a7d8da4-e26f-4ed1-852d-6f768a50450d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused"
Oct 30 16:37:11.598037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.597973    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:37:11.635046 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635015    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-dshm\") pod \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") "
Oct 30 16:37:11.635248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635228    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-model-cache\") pod \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") "
Oct 30 16:37:11.635403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635385    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7d8da4-e26f-4ed1-852d-6f768a50450d-kube-api-access-6htn5\" (UniqueName: \"kubernetes.io/projected/8a7d8da4-e26f-4ed1-852d-6f768a50450d-kube-api-access-6htn5\") pod \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") "
Oct 30 16:37:11.635500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635429    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tmp-dir\") pod \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") "
Oct 30 16:37:11.635500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635487    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-home\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-home\") pod \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") "
Oct 30 16:37:11.635609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635510    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-model-cache" pod "8a7d8da4-e26f-4ed1-852d-6f768a50450d" (UID: "8a7d8da4-e26f-4ed1-852d-6f768a50450d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:37:11.635609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635550    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tls-certs\") pod \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\" (UID: \"8a7d8da4-e26f-4ed1-852d-6f768a50450d\") "
Oct 30 16:37:11.635826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635766    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-home" pod "8a7d8da4-e26f-4ed1-852d-6f768a50450d" (UID: "8a7d8da4-e26f-4ed1-852d-6f768a50450d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:37:11.635826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.635810    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tmp-dir" pod "8a7d8da4-e26f-4ed1-852d-6f768a50450d" (UID: "8a7d8da4-e26f-4ed1-852d-6f768a50450d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:37:11.636095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.636074    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:11.636160 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.636100    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:11.636160 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.636113    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:11.639701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.639639    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-dshm" pod "8a7d8da4-e26f-4ed1-852d-6f768a50450d" (UID: "8a7d8da4-e26f-4ed1-852d-6f768a50450d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:37:11.639872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.639845    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tls-certs" pod "8a7d8da4-e26f-4ed1-852d-6f768a50450d" (UID: "8a7d8da4-e26f-4ed1-852d-6f768a50450d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:37:11.640001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.639973    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7d8da4-e26f-4ed1-852d-6f768a50450d-kube-api-access-6htn5" pod "8a7d8da4-e26f-4ed1-852d-6f768a50450d" (UID: "8a7d8da4-e26f-4ed1-852d-6f768a50450d"). InnerVolumeSpecName "kube-api-access-6htn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:37:11.736909 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.736875    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6htn5\" (UniqueName: \"kubernetes.io/projected/8a7d8da4-e26f-4ed1-852d-6f768a50450d-kube-api-access-6htn5\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:11.736909 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.736903    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7d8da4-e26f-4ed1-852d-6f768a50450d-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:11.736909 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.736913    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a7d8da4-e26f-4ed1-852d-6f768a50450d-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:11.921517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.921432    4497 generic.go:356] "Generic (PLEG): container finished" podID="8a7d8da4-e26f-4ed1-852d-6f768a50450d" containerID="6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3" exitCode=0
Oct 30 16:37:11.921517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.921482    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"
Oct 30 16:37:11.921721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.921515    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf" event={"ID":"8a7d8da4-e26f-4ed1-852d-6f768a50450d","Type":"ContainerDied","Data":"6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3"}
Oct 30 16:37:11.921721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.921557    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf" event={"ID":"8a7d8da4-e26f-4ed1-852d-6f768a50450d","Type":"ContainerDied","Data":"063c561fcca25616e2e1e06c2b931c634135aa8da2ab24a8a2dd394c87675f6a"}
Oct 30 16:37:11.921721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.921575    4497 scope.go:122] "RemoveContainer" containerID="6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3"
Oct 30 16:37:11.930967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.930950    4497 scope.go:122] "RemoveContainer" containerID="6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3"
Oct 30 16:37:11.931283 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:37:11.931254    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3\": container with ID starting with 6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3 not found: ID does not exist" containerID="6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3"
Oct 30 16:37:11.931389 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.931289    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3"} err="failed to get container status \"6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3\": rpc error: code = NotFound desc = could not find container \"6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3\": container with ID starting with 6ab2a9b370e779dbd3e11d78e51f5e248080fe68f79ede114b5e3e0af48623e3 not found: ID does not exist"
Oct 30 16:37:11.965838 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.965747    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"]
Oct 30 16:37:11.975700 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:11.975674    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-b-239c91ff-kserve-575dc89c86-cfjgf"]
Oct 30 16:37:12.299879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.299849    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7d8da4-e26f-4ed1-852d-6f768a50450d" path="/var/lib/kubelet/pods/8a7d8da4-e26f-4ed1-852d-6f768a50450d/volumes"
Oct 30 16:37:12.951108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.951069    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"]
Oct 30 16:37:12.951630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.951616    4497 cpu_manager.go:414] "containerMap: removing container" podUID="8a7d8da4-e26f-4ed1-852d-6f768a50450d" containerName="main"
Oct 30 16:37:12.951674 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.951632    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7d8da4-e26f-4ed1-852d-6f768a50450d" containerName="main"
Oct 30 16:37:12.951714 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.951706    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7d8da4-e26f-4ed1-852d-6f768a50450d" containerName="main"
Oct 30 16:37:12.956264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.956240    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:12.971907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.971879    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"]
Oct 30 16:37:12.975601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.975582    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:12.982778 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:12.982744    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"]
Oct 30 16:37:13.026804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.026769    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"]
Oct 30 16:37:13.052565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052532    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.052755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052570    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59cf364-b247-4075-add6-def59435ea16-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.052755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052598    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.052755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052627    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.052933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052800    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0814617-855c-462f-86f0-bfecf957544d-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.052933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052845    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0814617-855c-462f-86f0-bfecf957544d-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.052933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052925    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.053098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052954    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2sg\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-kube-api-access-9p2sg\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.053098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.052979    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.053162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053112    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.053162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053155    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.053247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053229    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42z7l\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-kube-api-access-42z7l\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.053285 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053261    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59cf364-b247-4075-add6-def59435ea16-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.053317 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053291    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.053374 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053361    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.053423 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053389    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.053460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053430    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.053460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.053448    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154233    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59cf364-b247-4075-add6-def59435ea16-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.154460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154283    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.154460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154320    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.154460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154346    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154368    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.154460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154390    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154441    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154464    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59cf364-b247-4075-add6-def59435ea16-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154491    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154521    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154579    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0814617-855c-462f-86f0-bfecf957544d-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154601    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0814617-855c-462f-86f0-bfecf957544d-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154650    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154674    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2sg\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-kube-api-access-9p2sg\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154699    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154748    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.154788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154764    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.155375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154775    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.155375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154813    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.155375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.154890    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42z7l\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-kube-api-access-42z7l\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.155375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.155117    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.157659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.157383    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.157895 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.157676    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.157895 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.157727    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.158022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.157962    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0814617-855c-462f-86f0-bfecf957544d-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.158022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.157973    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.158571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.158146    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.158571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.158516    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59cf364-b247-4075-add6-def59435ea16-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.158854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.158825    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.158977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.158891    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59cf364-b247-4075-add6-def59435ea16-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.159057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.159037    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0814617-855c-462f-86f0-bfecf957544d-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.160044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.160020    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.175711 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.175683    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.175866 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.175686    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.176200 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.176168    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2sg\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-kube-api-access-9p2sg\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.177258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.177236    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42z7l\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-kube-api-access-42z7l\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.270198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.270160    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:13.290603 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.290522    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:13.612078 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.611985    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"]
Oct 30 16:37:13.615202 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.615161    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"]
Oct 30 16:37:13.617165 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:37:13.617136    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59cf364_b247_4075_add6_def59435ea16.slice/crio-2dac896e67a5b33059e97774644680131787de1753ebf406357c06da7299c2a4 WatchSource:0}: Error finding container 2dac896e67a5b33059e97774644680131787de1753ebf406357c06da7299c2a4: Status 404 returned error can't find the container with id 2dac896e67a5b33059e97774644680131787de1753ebf406357c06da7299c2a4
Oct 30 16:37:13.619736 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.619699    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:13.619833 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.619778    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:13.619833 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.619826    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:13.630369 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:37:13.630346    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0814617_855c_462f_86f0_bfecf957544d.slice/crio-b446365958c72f7d2a9be295d6b6f6b078c614dd9e5dbd4451335fc70400a514 WatchSource:0}: Error finding container b446365958c72f7d2a9be295d6b6f6b078c614dd9e5dbd4451335fc70400a514: Status 404 returned error can't find the container with id b446365958c72f7d2a9be295d6b6f6b078c614dd9e5dbd4451335fc70400a514
Oct 30 16:37:13.632732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.632698    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:13.632833 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.632778    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:13.632833 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.632823    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:13.937759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.937673    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" event={"ID":"c0814617-855c-462f-86f0-bfecf957544d","Type":"ContainerStarted","Data":"18a0c549b8cbb596368af8e497ce4885177f08518dca75152857dcd376b345dc"}
Oct 30 16:37:13.937759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.937734    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" event={"ID":"c0814617-855c-462f-86f0-bfecf957544d","Type":"ContainerStarted","Data":"b446365958c72f7d2a9be295d6b6f6b078c614dd9e5dbd4451335fc70400a514"}
Oct 30 16:37:13.939502 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.939471    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" event={"ID":"c59cf364-b247-4075-add6-def59435ea16","Type":"ContainerStarted","Data":"f2ad4f6d448ab390bcca0e3424a705885e2cb92a154e6c901fdbd0d842e5300b"}
Oct 30 16:37:13.939635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.939515    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" event={"ID":"c59cf364-b247-4075-add6-def59435ea16","Type":"ContainerStarted","Data":"2dac896e67a5b33059e97774644680131787de1753ebf406357c06da7299c2a4"}
Oct 30 16:37:13.980489 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:13.980427    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podStartSLOduration=1.98040864 podStartE2EDuration="1.98040864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:13.976774118 +0000 UTC m=+863.830652387" watchObservedRunningTime="2026-10-30 16:37:13.98040864 +0000 UTC m=+863.834286906"
Oct 30 16:37:14.017740 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:14.017680    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podStartSLOduration=2.01765836 podStartE2EDuration="2.01765836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:14.016697957 +0000 UTC m=+863.870576222" watchObservedRunningTime="2026-10-30 16:37:14.01765836 +0000 UTC m=+863.871536627"
Oct 30 16:37:14.270863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:14.270830    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:14.272300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:14.272273    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 16:37:14.272424 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:14.272358    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 16:37:14.291268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:14.291232    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:14.292801 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:14.292776    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 16:37:14.292929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:14.292825    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 16:37:15.270880 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:15.270847    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 16:37:15.271327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:15.270915    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 16:37:15.293212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:15.292568    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 16:37:15.293212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:15.292637    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 16:37:16.270916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:16.270882    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 16:37:16.274932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:16.274512    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 16:37:16.292825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:16.292695    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 16:37:16.292825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:16.292759    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 16:37:17.279223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:17.271223    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 16:37:17.288572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:17.288532    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 16:37:17.307379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:17.307340    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 16:37:17.307649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:17.307616    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 16:37:17.888211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:17.876322    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"]
Oct 30 16:37:17.902496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:17.902463    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.009219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:17.999973    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.009583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.009548    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.009759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.009741    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.009955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.009937    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tq7\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-kube-api-access-x9tq7\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.014214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.010101    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.014417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.014375    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444f234d-59b5-4fc5-b736-2fa11c30aa91-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.014579 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.014563    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.015211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.014738    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.022475 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.022443    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.098763 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.098319    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"]
Oct 30 16:37:18.167306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.141424    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283166    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283272    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283329    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283404    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tq7\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-kube-api-access-x9tq7\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283471    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283499    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444f234d-59b5-4fc5-b736-2fa11c30aa91-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283545    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283637    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.284576    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.283092    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.284651    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.285486    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.285738    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.291223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.286677    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.316219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.315761    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444f234d-59b5-4fc5-b736-2fa11c30aa91-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.342317 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.339374    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tq7\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-kube-api-access-x9tq7\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.342555 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.340291    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 16:37:18.342707 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.342684    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 16:37:18.343139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.340959    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:18.486724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.486563    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.511645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.511606    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.512694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.512667    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.537359 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.537313    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:18.773357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.773320    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:18.882232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:18.873259    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 16:37:19.282845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.281973    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 16:37:19.282845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.282037    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 16:37:19.295787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.295633    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 16:37:19.295787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.295698    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 16:37:19.410986 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.408306    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:19.410986 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.409025    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:37:19.565213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.556528    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main" containerID="cri-o://0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511" gracePeriod=45
Oct 30 16:37:19.565213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.556662    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:19.565213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.556927    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:19.565213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.557333    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:37:19.815226 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.809453    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": EOF"
Oct 30 16:37:19.861319 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.860470    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:37:19.883998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.883390    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:19.886291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.884717    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:37:19.993435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:19.993127    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:37:20.026022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.025684    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"]
Oct 30 16:37:20.142787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.141787    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:20.142787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.141897    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:20.142787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.141972    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:20.322273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.298081    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 16:37:20.322273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.298148    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 16:37:20.322273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.298264    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 16:37:20.322273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.298288    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 16:37:20.543978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.543936    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:37:20.656258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.656142    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-home\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-home\") pod \"0fbf56dd-e248-43b3-b429-27705650b1a7\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") "
Oct 30 16:37:20.656258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.656225    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-tmp-dir\") pod \"0fbf56dd-e248-43b3-b429-27705650b1a7\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") "
Oct 30 16:37:20.656441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.656388    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-dshm\") pod \"0fbf56dd-e248-43b3-b429-27705650b1a7\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") "
Oct 30 16:37:20.656481 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.656433    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-model-cache\") pod \"0fbf56dd-e248-43b3-b429-27705650b1a7\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") "
Oct 30 16:37:20.658059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.656541    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbf56dd-e248-43b3-b429-27705650b1a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbf56dd-e248-43b3-b429-27705650b1a7-tls-certs\") pod \"0fbf56dd-e248-43b3-b429-27705650b1a7\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") "
Oct 30 16:37:20.658059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.656583    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbf56dd-e248-43b3-b429-27705650b1a7-kube-api-access-lq8km\" (UniqueName: \"kubernetes.io/projected/0fbf56dd-e248-43b3-b429-27705650b1a7-kube-api-access-lq8km\") pod \"0fbf56dd-e248-43b3-b429-27705650b1a7\" (UID: \"0fbf56dd-e248-43b3-b429-27705650b1a7\") "
Oct 30 16:37:20.661370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.660405    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-model-cache" pod "0fbf56dd-e248-43b3-b429-27705650b1a7" (UID: "0fbf56dd-e248-43b3-b429-27705650b1a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:37:20.661370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.660321    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-tmp-dir" pod "0fbf56dd-e248-43b3-b429-27705650b1a7" (UID: "0fbf56dd-e248-43b3-b429-27705650b1a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:37:20.661370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.660974    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-home" pod "0fbf56dd-e248-43b3-b429-27705650b1a7" (UID: "0fbf56dd-e248-43b3-b429-27705650b1a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:37:20.661733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.661707    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-dshm" pod "0fbf56dd-e248-43b3-b429-27705650b1a7" (UID: "0fbf56dd-e248-43b3-b429-27705650b1a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:37:20.665771 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.664774    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbf56dd-e248-43b3-b429-27705650b1a7-kube-api-access-lq8km" pod "0fbf56dd-e248-43b3-b429-27705650b1a7" (UID: "0fbf56dd-e248-43b3-b429-27705650b1a7"). InnerVolumeSpecName "kube-api-access-lq8km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:37:20.666460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.666439    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbf56dd-e248-43b3-b429-27705650b1a7-tls-certs" pod "0fbf56dd-e248-43b3-b429-27705650b1a7" (UID: "0fbf56dd-e248-43b3-b429-27705650b1a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:37:20.760207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.759923    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq8km\" (UniqueName: \"kubernetes.io/projected/0fbf56dd-e248-43b3-b429-27705650b1a7-kube-api-access-lq8km\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:20.760207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.760070    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:20.760207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.760091    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:20.760207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.760104    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:20.760207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.760117    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fbf56dd-e248-43b3-b429-27705650b1a7-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:20.760207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.760138    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fbf56dd-e248-43b3-b429-27705650b1a7-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:37:20.768272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.767436    4497 generic.go:356] "Generic (PLEG): container finished" podID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerID="0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511" exitCode=0
Oct 30 16:37:20.768272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.767491    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"
Oct 30 16:37:20.768272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.767632    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" event={"ID":"0fbf56dd-e248-43b3-b429-27705650b1a7","Type":"ContainerDied","Data":"0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511"}
Oct 30 16:37:20.768272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.767673    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" event={"ID":"0fbf56dd-e248-43b3-b429-27705650b1a7","Type":"ContainerDied","Data":"a094ac2e017075b5942bf962c2e72518400335f3778bf357f50efca47377b522"}
Oct 30 16:37:20.768272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.767690    4497 scope.go:122] "RemoveContainer" containerID="0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511"
Oct 30 16:37:20.808239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.805877    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" event={"ID":"444f234d-59b5-4fc5-b736-2fa11c30aa91","Type":"ContainerStarted","Data":"1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71"}
Oct 30 16:37:20.808239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.805961    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" event={"ID":"444f234d-59b5-4fc5-b736-2fa11c30aa91","Type":"ContainerStarted","Data":"2626f9f0b42a26436ad6f66ab893ab0b4a26b46b28dc9b8ec804be85f6cd228c"}
Oct 30 16:37:20.812655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.812622    4497 scope.go:122] "RemoveContainer" containerID="0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511"
Oct 30 16:37:20.813032 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:37:20.812945    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511\": container with ID starting with 0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511 not found: ID does not exist" containerID="0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511"
Oct 30 16:37:20.813032 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.812973    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511"} err="failed to get container status \"0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511\": rpc error: code = NotFound desc = could not find container \"0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511\": container with ID starting with 0c3609acb742c8208eb2cb9d7687c9ba53d86483e41a879e749bc9f651509511 not found: ID does not exist"
Oct 30 16:37:20.850022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.849963    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" podStartSLOduration=3.849945553 podStartE2EDuration="3.849945553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:20.847000709 +0000 UTC m=+870.700878974" watchObservedRunningTime="2026-10-30 16:37:20.849945553 +0000 UTC m=+870.703823819"
Oct 30 16:37:20.903913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.903874    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"]
Oct 30 16:37:20.908485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:20.908415    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f"]
Oct 30 16:37:21.270669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:21.270623    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 16:37:21.270879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:21.270704    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 16:37:21.301150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:21.301108    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 16:37:21.301460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:21.301431    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 16:37:21.443649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:21.443599    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/e2e-conflict-a-239c91ff-kserve-5ffffb4c68-tg75f" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:21.803646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:21.803566    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:21.809309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:21.808993    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 16:37:21.809309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:21.809050    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 16:37:21.937261 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:37:21.937215    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:37:22.321218 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.319561    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" path="/var/lib/kubelet/pods/0fbf56dd-e248-43b3-b429-27705650b1a7/volumes"
Oct 30 16:37:22.340759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.336903    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"]
Oct 30 16:37:22.340759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.337513    4497 cpu_manager.go:414] "containerMap: removing container" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main"
Oct 30 16:37:22.340759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.337533    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main"
Oct 30 16:37:22.340759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.337672    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbf56dd-e248-43b3-b429-27705650b1a7" containerName="main"
Oct 30 16:37:22.344544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.344520    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.358069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.358029    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"default-dockercfg-rzlfm\""
Oct 30 16:37:22.367785 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.367751    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"distinct-w2-18519c-kserve-self-signed-certs\""
Oct 30 16:37:22.367956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.367833    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"kube-root-ca.crt\""
Oct 30 16:37:22.370157 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.370131    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"openshift-service-ca.crt\""
Oct 30 16:37:22.418213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.415668    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"]
Oct 30 16:37:22.508366 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.508279    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-tmp-dir\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.508366 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.508345    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-dshm\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.508948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.508400    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-model-cache\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.508948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.508464    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea83974-ad90-4136-bce6-f0e4eecce09a-tls-certs\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.508948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.508572    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvst\" (UniqueName: \"kubernetes.io/projected/2ea83974-ad90-4136-bce6-f0e4eecce09a-kube-api-access-dnvst\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.508948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.508646    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-home\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.610176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.610076    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-home\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.610874 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.610655    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-home\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.611121 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.611102    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-tmp-dir\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.611675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.611657    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-dshm\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.616465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.611833    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-model-cache\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.617033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.611600    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-tmp-dir\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.617033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.612801    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-model-cache\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.617737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.617717    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea83974-ad90-4136-bce6-f0e4eecce09a-tls-certs\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.618453 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.618433    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvst\" (UniqueName: \"kubernetes.io/projected/2ea83974-ad90-4136-bce6-f0e4eecce09a-kube-api-access-dnvst\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.623083 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.623024    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea83974-ad90-4136-bce6-f0e4eecce09a-tls-certs\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.659210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.650158    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvst\" (UniqueName: \"kubernetes.io/projected/2ea83974-ad90-4136-bce6-f0e4eecce09a-kube-api-access-dnvst\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.659210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.657478    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-dshm\") pod \"distinct-w2-18519c-kserve-79d5b86f96-894h6\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.663209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.661935    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:22.816771 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.797668    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 16:37:22.816771 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:22.797731    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 16:37:23.059509 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.059467    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"]
Oct 30 16:37:23.065220 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:37:23.064615    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea83974_ad90_4136_bce6_f0e4eecce09a.slice/crio-8eaa17778690d0cb0da7a65e62561e32d9c888c91663e403647d74f381efd6b7 WatchSource:0}: Error finding container 8eaa17778690d0cb0da7a65e62561e32d9c888c91663e403647d74f381efd6b7: Status 404 returned error can't find the container with id 8eaa17778690d0cb0da7a65e62561e32d9c888c91663e403647d74f381efd6b7
Oct 30 16:37:23.316136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.315578    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:37:23.316136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.315643    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:23.316761 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.316493    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:37:23.316761 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.316540    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:23.799636 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.799595    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 16:37:23.800288 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.800249    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 16:37:23.851210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.850401    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" event={"ID":"2ea83974-ad90-4136-bce6-f0e4eecce09a","Type":"ContainerStarted","Data":"28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448"}
Oct 30 16:37:23.851210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.850453    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" event={"ID":"2ea83974-ad90-4136-bce6-f0e4eecce09a","Type":"ContainerStarted","Data":"8eaa17778690d0cb0da7a65e62561e32d9c888c91663e403647d74f381efd6b7"}
Oct 30 16:37:23.937472 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.936681    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podStartSLOduration=1.936667616 podStartE2EDuration="1.936667616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:23.935612157 +0000 UTC m=+873.789490422" watchObservedRunningTime="2026-10-30 16:37:23.936667616 +0000 UTC m=+873.790545882"
Oct 30 16:37:23.969885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.969852    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:23.970165 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.970143    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:23.997212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:23.996517    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:37:24.360422 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:24.360372    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:37:24.360632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:24.360442    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:24.506056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:24.506024    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:24.506498 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:24.506477    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:24.507347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:24.507329    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:37:24.662171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:24.662071    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:24.797781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:24.797738    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 16:37:24.797992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:24.797813    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 16:37:25.797566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:25.797529    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 16:37:25.798125 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:25.797591    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 16:37:26.797503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:26.797465    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 16:37:26.797688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:26.797530    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 16:37:27.801730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:27.801699    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:27.802144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:27.802056    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:27.802842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:27.802823    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:37:28.061811 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.061718    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"]
Oct 30 16:37:28.070497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.070464    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.096805 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.096752    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"]
Oct 30 16:37:28.127649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.127612    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.127649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.127653    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.127883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.127693    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.127883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.127845    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1db647c-9e15-447c-b59a-e9a40d2fa443-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.127883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.127876    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.128008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.127978    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.128008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.127996    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.128095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.128046    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.128095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.128077    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4vt\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-kube-api-access-rg4vt\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229413 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229373    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1db647c-9e15-447c-b59a-e9a40d2fa443-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229428    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229506    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229532    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229561    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229591    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4vt\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-kube-api-access-rg4vt\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229664    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229698    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.229869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229813    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.230084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.229993    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.230084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.230061    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.230240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.230171    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.230312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.230279    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.230636 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.230614    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1db647c-9e15-447c-b59a-e9a40d2fa443-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.232140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.232101    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.232636 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.232616    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.250470 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.250435    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.250590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.250567    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4vt\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-kube-api-access-rg4vt\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.391599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.391502    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:28.722668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.722639    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"]
Oct 30 16:37:28.725532 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:37:28.725500    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1db647c_9e15_447c_b59a_e9a40d2fa443.slice/crio-bb8c28180e3a389fcad10589ee87d4b298111f03012eac0689885e859a79c3ff WatchSource:0}: Error finding container bb8c28180e3a389fcad10589ee87d4b298111f03012eac0689885e859a79c3ff: Status 404 returned error can't find the container with id bb8c28180e3a389fcad10589ee87d4b298111f03012eac0689885e859a79c3ff
Oct 30 16:37:28.728062 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.728028    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:28.728140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.728094    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:28.728140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.728122    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:28.881060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.881016    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" event={"ID":"d1db647c-9e15-447c-b59a-e9a40d2fa443","Type":"ContainerStarted","Data":"7d4186ee696250802deb67ec5236679259be096cc5e872dd47939e24b5bf3a01"}
Oct 30 16:37:28.881578 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.881074    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" event={"ID":"d1db647c-9e15-447c-b59a-e9a40d2fa443","Type":"ContainerStarted","Data":"bb8c28180e3a389fcad10589ee87d4b298111f03012eac0689885e859a79c3ff"}
Oct 30 16:37:28.922654 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:28.922600    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podStartSLOduration=0.922584697 podStartE2EDuration="922.584697ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:28.922111958 +0000 UTC m=+878.775990224" watchObservedRunningTime="2026-10-30 16:37:28.922584697 +0000 UTC m=+878.776462962"
Oct 30 16:37:29.356451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.356409    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"]
Oct 30 16:37:29.360664 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.360633    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.390208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.388935    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"]
Oct 30 16:37:29.405210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.395006    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:29.412680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.412636    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 16:37:29.412885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.412701    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 16:37:29.448017 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.447963    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.448236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.448039    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.448236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.448117    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.448236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.448146    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.448431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.448315    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.448431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.448346    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3495-7f0c-4435-b244-de4fa133b035-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.448542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.448497    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx7m2\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-kube-api-access-gx7m2\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.448542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.448530    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.448652 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.448634    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.549616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549577    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.549804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549635    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.549804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549668    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.549804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549692    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.549804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549763    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.549804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549785    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3495-7f0c-4435-b244-de4fa133b035-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.550054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549895    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx7m2\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-kube-api-access-gx7m2\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.550054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549924    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.550054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.549999    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.550321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.550147    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.550321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.550166    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.550570 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.550547    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.550655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.550575    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3495-7f0c-4435-b244-de4fa133b035-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.550710 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.550643    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.552043 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.552024    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.552526 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.552509    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.570851 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.570813    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.571005 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.570942    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx7m2\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-kube-api-access-gx7m2\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:29.695295 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:29.694443    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:30.025108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:30.024881    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"]
Oct 30 16:37:30.027704 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:37:30.027679    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3e3495_7f0c_4435_b244_de4fa133b035.slice/crio-7e280bfea07e51701c712e7aa2b9332ceefaf722e8e8cc04524b7327d5ac560c WatchSource:0}: Error finding container 7e280bfea07e51701c712e7aa2b9332ceefaf722e8e8cc04524b7327d5ac560c: Status 404 returned error can't find the container with id 7e280bfea07e51701c712e7aa2b9332ceefaf722e8e8cc04524b7327d5ac560c
Oct 30 16:37:30.030222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:30.030181    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:30.030296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:30.030258    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:30.030296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:30.030288    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:30.402267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:30.395594    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 16:37:30.402267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:30.395656    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 16:37:30.928264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:30.927453    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" event={"ID":"7f3e3495-7f0c-4435-b244-de4fa133b035","Type":"ContainerStarted","Data":"01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886"}
Oct 30 16:37:30.928264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:30.927520    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" event={"ID":"7f3e3495-7f0c-4435-b244-de4fa133b035","Type":"ContainerStarted","Data":"7e280bfea07e51701c712e7aa2b9332ceefaf722e8e8cc04524b7327d5ac560c"}
Oct 30 16:37:31.421240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:31.420306    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 16:37:31.421240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:31.420374    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 16:37:31.699263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:31.698888    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:31.707207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:31.706327    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 16:37:31.707207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:31.706386    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 16:37:32.392692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:32.392647    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 16:37:32.394455 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:32.394420    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 16:37:32.668502 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:32.668408    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:32.709483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:32.709444    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 16:37:32.709781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:32.709748    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 16:37:32.923274 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:32.917304    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:33.394669 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:33.394634    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 16:37:33.394973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:33.394940    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 16:37:33.703664 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:33.703581    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 16:37:33.704282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:33.704250    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 16:37:33.909349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:33.909301    4497 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:33.951323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:33.951275    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:33.951858 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:33.951734    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:37:34.041218 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:34.035722    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podStartSLOduration=5.0356998 podStartE2EDuration="5.0356998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:31.006246714 +0000 UTC m=+880.860125011" watchObservedRunningTime="2026-10-30 16:37:34.0356998 +0000 UTC m=+883.889578066"
Oct 30 16:37:34.267842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:34.267814    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:37:34.404311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:34.400614    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 16:37:34.404311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:34.400674    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 16:37:34.703412 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:34.703314    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 16:37:34.703662 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:34.703632    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 16:37:35.401066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.401026    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 16:37:35.401679 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.401644    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 16:37:35.721518 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.708286    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 16:37:35.721518 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.708350    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 16:37:35.742338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.742303    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"]
Oct 30 16:37:35.747286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.747262    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.766396 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.766359    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"]
Oct 30 16:37:35.827622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.827590    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"distinct-2-w4-420ff3-kserve-self-signed-certs\""
Oct 30 16:37:35.828921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.828895    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-model-cache\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.829142 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.829120    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tmp-dir\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.829326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.829308    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-home\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.829623 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.829601    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tls-certs\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.829815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.829779    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4wd\" (UniqueName: \"kubernetes.io/projected/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-kube-api-access-xq4wd\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.830051 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.830036    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-dshm\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.931656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.931605    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-dshm\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.931830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.931729    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-model-cache\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.931830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.931753    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tmp-dir\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.931830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.931790    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-home\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.931939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.931915    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tls-certs\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.931985 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.931970    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4wd\" (UniqueName: \"kubernetes.io/projected/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-kube-api-access-xq4wd\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.932490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.932456    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tmp-dir\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.932490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.932469    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-model-cache\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.934161 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.934136    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-home\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.934284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.934139    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-dshm\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.934548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.934529    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tls-certs\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:35.954127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:35.954086    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4wd\" (UniqueName: \"kubernetes.io/projected/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-kube-api-access-xq4wd\") pod \"distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:36.062551 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:36.062516    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:36.394264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:36.393605    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 16:37:36.394264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:36.393684    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 16:37:36.447634 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:36.447607    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"]
Oct 30 16:37:36.705820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:36.705723    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 16:37:36.705820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:36.705797    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 16:37:37.437779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:37.437743    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" event={"ID":"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f","Type":"ContainerStarted","Data":"f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd"}
Oct 30 16:37:37.437998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:37.437799    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" event={"ID":"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f","Type":"ContainerStarted","Data":"4e0e088360c0799f23d0d26ec596fd01654045432ee8f27e0e7fb1cce2f28fe8"}
Oct 30 16:37:37.450980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:37.450948    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:37.451453 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:37.451428    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:37.452515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:37.452496    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:37:37.479801 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:37.479744    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podStartSLOduration=2.4797314520000002 podStartE2EDuration="2.479731452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:37.47837784 +0000 UTC m=+887.332256109" watchObservedRunningTime="2026-10-30 16:37:37.479731452 +0000 UTC m=+887.333609716"
Oct 30 16:37:37.696268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:37.696147    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 16:37:37.696268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:37.696253    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 16:37:38.063068 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:38.063012    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:39.659640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:39.659593    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:39.660144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:39.660119    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:39.661292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:39.661270    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:37:42.772221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:42.771365    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:43.071270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:43.070578    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:37:46.062854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:46.062815    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:46.075476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:46.075446    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:46.530567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:46.530539    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:51.965824 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:37:51.965784    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:37:53.521356 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:53.521300    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:37:53.679811 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:53.678838    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:53.706989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:53.705969    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:53.791264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:53.791149    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:53.997326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:53.997280    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 16:37:54.289254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:54.278306    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:54.289254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:54.278801    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:37:54.289254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:54.278844    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:54.537413 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:54.537349    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:54.539890 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:54.538733    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:54.794320 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:54.777302    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:54.794320 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:54.778117    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:37:55.197065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:55.196967    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:37:55.197348 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:55.197317    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:37:55.324508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:55.324475    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:37:55.391355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:55.389610    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:37:57.470405 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.470371    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"]
Oct 30 16:37:57.475982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.475962    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.488983 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.488963    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a77b48-openshift-default-dockercfg-725dg\""
Oct 30 16:37:57.502221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.502172    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"]
Oct 30 16:37:57.597567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.597523    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.597567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.597573    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.597846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.597614    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.597846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.597731    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z256k\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-kube-api-access-z256k\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.597846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.597824    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.598009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.597905    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.598009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.597953    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.598099 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.598067    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.598175 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.598158    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.699961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.699925    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.699961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.699959    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.699978    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z256k\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-kube-api-access-z256k\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700098    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700238    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700290    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700352    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700526 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700398    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700526 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700461    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700526 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700476    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700613    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700655    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700771 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700726    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.700977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.700954    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.702638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.702612    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.702715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.702665    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.721478 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.721408    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.721782 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.721751    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z256k\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-kube-api-access-z256k\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:57.789857 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:57.789827    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:58.113128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:58.113103    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"]
Oct 30 16:37:58.116094 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:37:58.116065    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6c7303_9e9a_4a77_870d_548e12b8dfdf.slice/crio-9adf47f5f78f44a9f316ab5a5fc29f0028fa7ea4fa0cce8b76946b6449b9e741 WatchSource:0}: Error finding container 9adf47f5f78f44a9f316ab5a5fc29f0028fa7ea4fa0cce8b76946b6449b9e741: Status 404 returned error can't find the container with id 9adf47f5f78f44a9f316ab5a5fc29f0028fa7ea4fa0cce8b76946b6449b9e741
Oct 30 16:37:58.118536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:58.118503    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:58.118622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:58.118569    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:58.118622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:58.118604    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:37:58.233995 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:58.233962    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" event={"ID":"bd6c7303-9e9a-4a77-870d-548e12b8dfdf","Type":"ContainerStarted","Data":"e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5"}
Oct 30 16:37:58.234182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:58.234023    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" event={"ID":"bd6c7303-9e9a-4a77-870d-548e12b8dfdf","Type":"ContainerStarted","Data":"9adf47f5f78f44a9f316ab5a5fc29f0028fa7ea4fa0cce8b76946b6449b9e741"}
Oct 30 16:37:58.283057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:58.282979    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podStartSLOduration=1.282959459 podStartE2EDuration="1.282959459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:58.279059408 +0000 UTC m=+908.132937673" watchObservedRunningTime="2026-10-30 16:37:58.282959459 +0000 UTC m=+908.136837724"
Oct 30 16:37:58.817486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:58.817450    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:37:59.796350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:59.796302    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 16:37:59.796547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:37:59.796387    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 16:38:00.794211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:00.793642    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 16:38:00.794211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:00.793707    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 16:38:01.795586 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:01.795547    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 16:38:01.796197 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:01.796146    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 16:38:02.793123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:02.792634    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 16:38:02.793123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:02.792714    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 16:38:03.796702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:03.796660    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 16:38:03.797131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:03.796751    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 16:38:04.816813 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:04.813404    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 16:38:04.816813 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:04.813476    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 16:38:06.008024 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.007204    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 16:38:06.271311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.270307    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:06.271311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.270417    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:06.337266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.335317    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:06.405869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.405816    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:06.419096 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.417592    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:38:06.419096 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.419031    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:38:06.423536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.423464    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:38:06.781375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:06.780690    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:10.671732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.671694    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"]
Oct 30 16:38:10.677316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.677289    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.701132 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.701095    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"default-dockercfg-pvgxh\""
Oct 30 16:38:10.701433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.701411    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"openshift-service-ca.crt\""
Oct 30 16:38:10.701611 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.701591    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"distinct-2-w1-a77b48-kserve-self-signed-certs\""
Oct 30 16:38:10.701732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.701716    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"kube-root-ca.crt\""
Oct 30 16:38:10.708083 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.708037    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"]
Oct 30 16:38:10.820681 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.820646    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79tgd\" (UniqueName: \"kubernetes.io/projected/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-kube-api-access-79tgd\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.820877 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.820710    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tmp-dir\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.820877 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.820796    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tls-certs\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.820968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.820914    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-model-cache\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.821065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.821048    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-home\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.821130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.821094    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-dshm\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.922580 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.922474    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-home\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.922580 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.922544    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-dshm\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.922580 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.922574    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79tgd\" (UniqueName: \"kubernetes.io/projected/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-kube-api-access-79tgd\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.922835 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.922653    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tmp-dir\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.922835 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.922687    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tls-certs\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.922914 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.922839    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-model-cache\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.923626 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.923574    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-home\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.924208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.924150    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-model-cache\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.924208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.924167    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tmp-dir\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.927471 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.927403    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tls-certs\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.936491 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.936459    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-dshm\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.945386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.945351    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79tgd\" (UniqueName: \"kubernetes.io/projected/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-kube-api-access-79tgd\") pod \"distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:10.998527 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:10.998484    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:11.307548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:11.307523    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"]
Oct 30 16:38:11.310541 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:38:11.310509    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c003d2c_031f_4b5e_a6dc_4c88bd8fa03f.slice/crio-3c9682c059aba300d8c9e0f8a61eb86b8643bb61159b6b1f35530f227c70bc42 WatchSource:0}: Error finding container 3c9682c059aba300d8c9e0f8a61eb86b8643bb61159b6b1f35530f227c70bc42: Status 404 returned error can't find the container with id 3c9682c059aba300d8c9e0f8a61eb86b8643bb61159b6b1f35530f227c70bc42
Oct 30 16:38:11.399095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:11.399050    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" event={"ID":"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f","Type":"ContainerStarted","Data":"36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5"}
Oct 30 16:38:11.399095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:11.399101    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" event={"ID":"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f","Type":"ContainerStarted","Data":"3c9682c059aba300d8c9e0f8a61eb86b8643bb61159b6b1f35530f227c70bc42"}
Oct 30 16:38:11.437246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:11.437180    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podStartSLOduration=1.43716461 podStartE2EDuration="1.43716461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:11.436412314 +0000 UTC m=+921.290290583" watchObservedRunningTime="2026-10-30 16:38:11.43716461 +0000 UTC m=+921.291042912"
Oct 30 16:38:11.999532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:11.999496    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:18.049026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.048989    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"]
Oct 30 16:38:18.053673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.053649    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.087054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.087014    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"]
Oct 30 16:38:18.223717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.223672    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.223717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.223723    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.223941 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.223863    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.223941 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.223897    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.224012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.223992    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.224062 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.224047    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65gpk\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-kube-api-access-65gpk\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.224167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.224155    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.224270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.224257    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.224307 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.224282    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218be99c-35a8-4d96-9726-edcc8a16bb42-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325320 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325240    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325320 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325278    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325338    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325400    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65gpk\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-kube-api-access-65gpk\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325503    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325582    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325617    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218be99c-35a8-4d96-9726-edcc8a16bb42-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325665    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325698    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325943 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325767    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.325943 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.325816    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.326089 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.326065    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.326162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.326118    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.326470 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.326443    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218be99c-35a8-4d96-9726-edcc8a16bb42-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.328003 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.327974    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.328522 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.328489    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.346562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.346526    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.346705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.346678    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65gpk\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-kube-api-access-65gpk\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.369294 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.369263    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:18.698425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.698256    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"]
Oct 30 16:38:18.701500 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:38:18.701468    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218be99c_35a8_4d96_9726_edcc8a16bb42.slice/crio-42f7a0eadc0df0f7dc49965f72f7b4332b56488cf437fdb1b0f1bd1081aeec95 WatchSource:0}: Error finding container 42f7a0eadc0df0f7dc49965f72f7b4332b56488cf437fdb1b0f1bd1081aeec95: Status 404 returned error can't find the container with id 42f7a0eadc0df0f7dc49965f72f7b4332b56488cf437fdb1b0f1bd1081aeec95
Oct 30 16:38:18.703934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.703899    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:38:18.704015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.703974    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:38:18.704015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:18.704002    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:38:19.462756 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:19.462718    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" event={"ID":"218be99c-35a8-4d96-9726-edcc8a16bb42","Type":"ContainerStarted","Data":"19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f"}
Oct 30 16:38:19.463163 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:19.462772    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" event={"ID":"218be99c-35a8-4d96-9726-edcc8a16bb42","Type":"ContainerStarted","Data":"42f7a0eadc0df0f7dc49965f72f7b4332b56488cf437fdb1b0f1bd1081aeec95"}
Oct 30 16:38:19.515560 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:19.515506    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podStartSLOduration=1.515490188 podStartE2EDuration="1.515490188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:19.512567004 +0000 UTC m=+929.366445270" watchObservedRunningTime="2026-10-30 16:38:19.515490188 +0000 UTC m=+929.369368453"
Oct 30 16:38:20.370271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:20.370231    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:20.372490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:20.372460    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:20.372726 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:20.372698    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:21.000009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:20.999644    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:21.083210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:21.082900    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:21.370109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:21.370073    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:21.370322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:21.370142    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:21.617640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:21.617610    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:22.372821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:22.372783    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:22.398213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:22.378686    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:22.486217 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:38:22.478497    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:38:23.126355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:23.126305    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:23.279506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:23.278307    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:23.330338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:23.329297    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 16:38:23.371018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:23.370593    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:23.371018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:23.370663    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:23.401587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:23.401467    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:23.499410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:23.499349    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:24.058214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.034483    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:24.133212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.132024    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 16:38:24.133212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.132666    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:38:24.294211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.290328    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:24.321429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.321339    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:38:24.375349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.329464    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:24.429372 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.428898    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:38:24.494243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.492417    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:24.494243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.492486    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:24.509244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.500257    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 16:38:24.546590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.544955    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:38:24.546590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.545026    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:38:24.546590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:24.545063    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:25.047212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.045051    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:25.047212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.045867    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:38:25.081410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.081360    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:38:25.085338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.085289    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:25.142259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.133887    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 16:38:25.309463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.309350    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 16:38:25.393759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.381867    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:25.394401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.393341    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:25.394625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.394586    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:25.440318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.440278    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:38:25.507664 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.506476    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:25.564273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.554555    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:38:25.564273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.554624    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:25.811924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.811887    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:38:25.812214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:25.812161    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:26.050416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.050369    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:26.204875 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.204827    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:26.322486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.322374    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 30 16:38:26.403849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.403788    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 16:38:26.405668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.405621    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:26.407305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.407265    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:26.407498 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.407476    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:26.447693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.447652    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:38:26.549831 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.549787    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:26.721035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.720929    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:38:26.726868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.726841    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:38:26.731927 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.731906    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:38:26.733892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.733872    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:38:26.740073 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.736814    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:26.743915 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:26.743895    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:38:27.370419 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:27.370382    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:27.370647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:27.370439    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:28.370245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:28.370174    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 16:38:28.370785 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:28.370274    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 16:38:29.374202 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:29.374156    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:29.376550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:29.374688    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:29.376550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:29.375503    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:38:38.744383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:38.744275    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:38.745324 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:38.745112    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:38.745864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:38.745147    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:38:38.746268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:38.745178    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:39.185932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.183291    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:39.396331 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.396278    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:39.503400 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.503286    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:39.747229 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.746331    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:39.765270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.747922    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 16:38:39.765270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.748041    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:39.765270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.748322    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:39.765270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.748725    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:38:39.765270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.748788    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:38:39.765270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.748826    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:39.765270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:39.748862    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:38:40.069268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.047304    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:38:40.069268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.047375    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:40.197213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.196195    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:40.208223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.197550    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:38:40.413420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.413294    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:40.504212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.502641    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:40.717634 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.717537    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:38:40.717985 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.717949    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:40.820255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.748272    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:40.820255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.748422    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:40.820255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.748485    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:40.820255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:40.756544    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:41.507933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:41.505341    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:42.024416 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:38:42.024379    4497 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.601s"
Oct 30 16:38:42.043432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.043401    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:38:42.044168 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.044139    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:38:42.539238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.539179    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 16:38:42.539839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.539806    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:38:42.540286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.540262    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 16:38:42.540661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.540637    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:38:42.541022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.540996    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 16:38:42.541407 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.541384    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Oct 30 16:38:42.541601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.541581    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 16:38:42.663220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:42.663148    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:38:43.001872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:43.001769    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:38:43.019999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:43.019558    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:38:43.021071 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:43.021042    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:38:43.022444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:43.022423    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:38:50.537909 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:38:50.537876    4497 scope.go:122] "RemoveContainer" containerID="ca64db6b529d688c008d3244261b9cd770576cf5c1d440be040ee2031f06dd9e"
Oct 30 16:38:51.916716 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:38:51.916673    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:39:02.492498 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.474047    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"]
Oct 30 16:39:02.838214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.832431    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"]
Oct 30 16:39:02.862322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.833532    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.904751    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-podinfo\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.904887    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-credential-socket\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.904922    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-certs\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.904963    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w22w\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-kube-api-access-9w22w\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.905042    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-token\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.905089    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-envoy\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.905152    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa339320-3f28-4d4a-9331-179ff100cdb1-istiod-ca-cert\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.905180    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-data\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.905260    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-socket\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:02.947849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:02.915453    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8295b2-openshift-default-dockercfg-cpfp5\""
Oct 30 16:39:03.019336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.019295    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-credential-socket\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.050983 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.050949    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-certs\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.051648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.051625    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w22w\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-kube-api-access-9w22w\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.052249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.052226    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-token\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.052693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.052674    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-envoy\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.053248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.053226    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa339320-3f28-4d4a-9331-179ff100cdb1-istiod-ca-cert\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.054081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.054061    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-data\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.054305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.054286    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-socket\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.054515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.054495    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-podinfo\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.054841 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.051545    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-certs\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.054967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.054015    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa339320-3f28-4d4a-9331-179ff100cdb1-istiod-ca-cert\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.067826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.035770    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:03.077926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.035720    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:03.079505 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.055561    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-socket\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.079685 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.027759    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-credential-socket\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.079794 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.056215    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-data\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.079884 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.067749    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-podinfo\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.123667 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.123587    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w22w\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-kube-api-access-9w22w\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.125391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.125357    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-envoy\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.126071 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.126026    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-token\") pod \"e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") " pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.168123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.168079    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 16:39:03.362505 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.362464    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:03.585990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.585292    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:03.644208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.643570    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 16:39:03.695364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.694785    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:03.784224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:03.783279    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:39:04.069376 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.069315    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:04.070593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.070560    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:39:04.085059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.085020    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 16:39:04.086600 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.085739    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:39:04.340274 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.316832    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 16:39:04.352232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.341347    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:39:04.602361 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.602258    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:04.660238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.652264    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 16:39:04.681221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.680380    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:39:04.681221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.680444    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:39:04.798213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.788299    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:39:04.881761 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:04.810808    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:39:05.108213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.083762    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:05.108940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.108897    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:05.222430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.212387    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:05.222430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.212675    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:05.225353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.224684    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:05.352706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.351402    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:05.560219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.544375    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"]
Oct 30 16:39:05.623452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.620492    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:05.689228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.653701    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 16:39:05.829911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.829830    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:05.835286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:05.835239    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" event={"ID":"aa339320-3f28-4d4a-9331-179ff100cdb1","Type":"ContainerStarted","Data":"7045545f2ea631adaacaa0f5c3e88bd105a32a4854326df109a9dbbdea900318"}
Oct 30 16:39:06.103273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.090333    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:06.146466 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.145272    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:06.351443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.351368    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 30 16:39:06.351815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.351783    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:06.408293 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.408250    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-pkh5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:39:06.452344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.452277    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:06.734213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.716920    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 16:39:06.847213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.705279    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:06.878311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.870269    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-p8zlx" podUID="2a524c7d-3802-443c-a1e0-056bc0e34af4" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:06.936271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:06.910915    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:39:07.108217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.085624    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:07.130994 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.130947    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:07.169238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.169114    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:07.362830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.362783    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:07.496328 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.496286    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:39:07.496666 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.496630    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:07.853209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.839972    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:39:08.067213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.061411    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:08.067213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.894001    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 16:39:08.067213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:07.893965    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 16:39:08.067213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.050012    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:08.084351 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.084309    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fr5gj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:39:08.084658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.084616    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:08.116706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.116595    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" event={"ID":"aa339320-3f28-4d4a-9331-179ff100cdb1","Type":"ContainerStarted","Data":"c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6"}
Oct 30 16:39:08.143728 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.137847    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:08.171325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.171272    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:08.172591 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.172565    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:39:08.174871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.174850    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 16:39:08.204090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.204059    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 16:39:08.255214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.251022    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" podStartSLOduration=6.251001207 podStartE2EDuration="6.251001207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:08.24713289 +0000 UTC m=+978.101011158" watchObservedRunningTime="2026-10-30 16:39:08.251001207 +0000 UTC m=+978.104879473"
Oct 30 16:39:08.459728 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.459639    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:08.956032 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.955995    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jzt7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:39:08.956644 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:08.956609    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jzt7r" podUID="c92b23e0-1211-499e-a41d-ab0ae8387659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:09.045257 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:09.045198    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:39:09.045591 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:09.045570    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:09.146900 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:09.146859    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:39:09.152851 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:09.152821    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:09.154598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:09.154577    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:09.172221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:09.158363    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 16:39:09.207654 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:09.206008    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 16:39:09.378968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:09.366419    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:39:15.170594 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.170559    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"]
Oct 30 16:39:15.175108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.175081    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.187641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.187611    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0b09f1-openshift-default-dockercfg-kfbk6\""
Oct 30 16:39:15.201697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.201664    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"]
Oct 30 16:39:15.257132 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257097    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-certs\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.257338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257146    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16e1d36-6550-473d-b77d-4cba06f6279d-istiod-ca-cert\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.257390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257363    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-token\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.257524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257506    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-socket\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.257646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257624    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-data\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.257715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257669    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-podinfo\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.257809 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257792    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-credential-socket\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.257871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257823    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ht6\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-kube-api-access-w5ht6\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.257871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.257853    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-envoy\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.358968    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-certs\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359021    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16e1d36-6550-473d-b77d-4cba06f6279d-istiod-ca-cert\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359111    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-token\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359307 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359234    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-socket\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359330    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-data\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359366    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-podinfo\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359423    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-credential-socket\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359551 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359453    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ht6\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-kube-api-access-w5ht6\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359551 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359468    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-certs\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359551 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359479    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-envoy\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359704 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359592    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-socket\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359711    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-credential-socket\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.359896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.359872    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16e1d36-6550-473d-b77d-4cba06f6279d-istiod-ca-cert\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.360049 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.360027    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-data\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.361706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.361681    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-envoy\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.362132 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.362110    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-podinfo\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.381776 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.381749    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-token\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.381907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.381838    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ht6\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-kube-api-access-w5ht6\") pod \"e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") " pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.490765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.490441    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:15.866829 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.866796    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"]
Oct 30 16:39:15.871505 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:39:15.871475    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16e1d36_6550_473d_b77d_4cba06f6279d.slice/crio-5ba75d26d0eee2f4ed1c7720b5af6708c259bfe5eb3065d08970bdf1f4eab97e WatchSource:0}: Error finding container 5ba75d26d0eee2f4ed1c7720b5af6708c259bfe5eb3065d08970bdf1f4eab97e: Status 404 returned error can't find the container with id 5ba75d26d0eee2f4ed1c7720b5af6708c259bfe5eb3065d08970bdf1f4eab97e
Oct 30 16:39:15.873886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.873854    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:15.873959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.873922    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:15.873959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:15.873951    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:16.198964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:16.198876    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" event={"ID":"a16e1d36-6550-473d-b77d-4cba06f6279d","Type":"ContainerStarted","Data":"603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25"}
Oct 30 16:39:16.198964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:16.198937    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" event={"ID":"a16e1d36-6550-473d-b77d-4cba06f6279d","Type":"ContainerStarted","Data":"5ba75d26d0eee2f4ed1c7720b5af6708c259bfe5eb3065d08970bdf1f4eab97e"}
Oct 30 16:39:16.241165 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:16.241099    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" podStartSLOduration=1.241079448 podStartE2EDuration="1.241079448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:16.239506521 +0000 UTC m=+986.093384790" watchObservedRunningTime="2026-10-30 16:39:16.241079448 +0000 UTC m=+986.094957714"
Oct 30 16:39:16.492541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:16.492451    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:16.498351 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:16.498325    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:17.205885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:17.205851    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:17.206978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:17.206942    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:39:20.636254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.636217    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"]
Oct 30 16:39:20.647703 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.647675    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:20.660446 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.660414    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"]
Oct 30 16:39:20.666434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.666412    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0b09f1-dockercfg-b9glg\""
Oct 30 16:39:20.675131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.675097    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0b09f1\""
Oct 30 16:39:20.755210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.754114    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p76ws\" (UniqueName: \"kubernetes.io/projected/60b23937-07d3-468f-a78b-987629d92056-kube-api-access-p76ws\") pod \"payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq\" (UID: \"60b23937-07d3-468f-a78b-987629d92056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:20.755210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.754378    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b23937-07d3-468f-a78b-987629d92056-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq\" (UID: \"60b23937-07d3-468f-a78b-987629d92056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:20.856141 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.856107    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b23937-07d3-468f-a78b-987629d92056-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq\" (UID: \"60b23937-07d3-468f-a78b-987629d92056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:20.856344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.856264    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p76ws\" (UniqueName: \"kubernetes.io/projected/60b23937-07d3-468f-a78b-987629d92056-kube-api-access-p76ws\") pod \"payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq\" (UID: \"60b23937-07d3-468f-a78b-987629d92056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:20.856876 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.856850    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b23937-07d3-468f-a78b-987629d92056-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq\" (UID: \"60b23937-07d3-468f-a78b-987629d92056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:20.877372 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.877341    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p76ws\" (UniqueName: \"kubernetes.io/projected/60b23937-07d3-468f-a78b-987629d92056-kube-api-access-p76ws\") pod \"payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq\" (UID: \"60b23937-07d3-468f-a78b-987629d92056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:20.963453 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:20.963362    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:21.272287 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:21.272115    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"]
Oct 30 16:39:21.275176 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:39:21.275149    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b23937_07d3_468f_a78b_987629d92056.slice/crio-4dd8eccb3a2318e3c8d7ab42881dfe2502c383bb7ce999a43f9f117d3e1c9506 WatchSource:0}: Error finding container 4dd8eccb3a2318e3c8d7ab42881dfe2502c383bb7ce999a43f9f117d3e1c9506: Status 404 returned error can't find the container with id 4dd8eccb3a2318e3c8d7ab42881dfe2502c383bb7ce999a43f9f117d3e1c9506
Oct 30 16:39:21.918417 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:39:21.918379    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:39:22.240709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:22.240604    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq" event={"ID":"60b23937-07d3-468f-a78b-987629d92056","Type":"ContainerStarted","Data":"69da91f9771b7359e14b04f4f2b3417c089e8a2164e73d23399b190514be854a"}
Oct 30 16:39:22.240709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:22.240669    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq" event={"ID":"60b23937-07d3-468f-a78b-987629d92056","Type":"ContainerStarted","Data":"4dd8eccb3a2318e3c8d7ab42881dfe2502c383bb7ce999a43f9f117d3e1c9506"}
Oct 30 16:39:22.240926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:22.240729    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:22.280204 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:22.280128    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq" podStartSLOduration=2.280107772 podStartE2EDuration="2.280107772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:22.277240544 +0000 UTC m=+992.131118806" watchObservedRunningTime="2026-10-30 16:39:22.280107772 +0000 UTC m=+992.133986038"
Oct 30 16:39:24.021510 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.021471    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"]
Oct 30 16:39:24.025822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.025801    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.042658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.042634    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8295b2-serving-cert\""
Oct 30 16:39:24.054538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.054444    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"]
Oct 30 16:39:24.102818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.102784    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-metrics-certs\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.103008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.102841    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldqf\" (UniqueName: \"kubernetes.io/projected/46ae9315-d893-4c58-b16b-e8cc6faf3045-kube-api-access-8ldqf\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.103101 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.103081    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae9315-d893-4c58-b16b-e8cc6faf3045-openshift-service-ca\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.103171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.103136    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-maas-api-tls\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.204293 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.204252    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-metrics-certs\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.204500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.204324    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldqf\" (UniqueName: \"kubernetes.io/projected/46ae9315-d893-4c58-b16b-e8cc6faf3045-kube-api-access-8ldqf\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.204500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.204438    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae9315-d893-4c58-b16b-e8cc6faf3045-openshift-service-ca\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.204500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.204493    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-maas-api-tls\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.205199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.205155    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae9315-d893-4c58-b16b-e8cc6faf3045-openshift-service-ca\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.207180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.207136    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-metrics-certs\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.207180 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.207272    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-maas-api-tls\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.225980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.225941    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldqf\" (UniqueName: \"kubernetes.io/projected/46ae9315-d893-4c58-b16b-e8cc6faf3045-kube-api-access-8ldqf\") pod \"maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.340245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.340125    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:24.665407 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.665380    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"]
Oct 30 16:39:24.668816 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:39:24.668791    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ae9315_d893_4c58_b16b_e8cc6faf3045.slice/crio-4f60e6009c9ee7056337883c9e3eeab0ca62837d0d0515d574db415ab419ea6f WatchSource:0}: Error finding container 4f60e6009c9ee7056337883c9e3eeab0ca62837d0d0515d574db415ab419ea6f: Status 404 returned error can't find the container with id 4f60e6009c9ee7056337883c9e3eeab0ca62837d0d0515d574db415ab419ea6f
Oct 30 16:39:24.670435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:24.670411    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:39:25.263269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:25.263239    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5" event={"ID":"46ae9315-d893-4c58-b16b-e8cc6faf3045","Type":"ContainerStarted","Data":"398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d"}
Oct 30 16:39:25.263672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:25.263288    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5" event={"ID":"46ae9315-d893-4c58-b16b-e8cc6faf3045","Type":"ContainerStarted","Data":"4f60e6009c9ee7056337883c9e3eeab0ca62837d0d0515d574db415ab419ea6f"}
Oct 30 16:39:25.263672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:25.263368    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:25.303329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:25.303275    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5" podStartSLOduration=1.890061029 podStartE2EDuration="2.303262372s" totalImagesPullingTime="413.201343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:25.302348828 +0000 UTC m=+995.156227094" watchObservedRunningTime="2026-10-30 16:39:25.303262372 +0000 UTC m=+995.157140637"
Oct 30 16:39:27.769492 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.769455    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"]
Oct 30 16:39:27.773919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.773897    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.802342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.802285    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"]
Oct 30 16:39:27.885879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.885840    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.886052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.885886    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r6j9\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-kube-api-access-4r6j9\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.886052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.886002    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.886052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.886038    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.886150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.886089    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.886203 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.886162    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.886290 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.886276    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.886330 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.886313    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.886417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.886405    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.987698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.987651    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6j9\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-kube-api-access-4r6j9\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.987698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.987709    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.987954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.987727    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.987954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.987747    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.987954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.987810    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.987954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.987844    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.987954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.987887    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.987954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.987950    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.988284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.988115    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.988284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.988274    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.988383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.988343    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.988648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.988620    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.988739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.988656    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.988800 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.988783    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.990233 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.990207    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:27.990339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:27.990322    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:28.014420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:28.014390    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:28.014561 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:28.014504    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r6j9\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-kube-api-access-4r6j9\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:28.092447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:28.092361    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:28.420113 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:28.420033    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"]
Oct 30 16:39:28.423784 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:39:28.423752    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b17bcb4_d397_4a7f_a067_a5fbf0265b1c.slice/crio-44e22a4b094b63b5a1077fc39d51054f53836d400b89c343c6a286343179abc3 WatchSource:0}: Error finding container 44e22a4b094b63b5a1077fc39d51054f53836d400b89c343c6a286343179abc3: Status 404 returned error can't find the container with id 44e22a4b094b63b5a1077fc39d51054f53836d400b89c343c6a286343179abc3
Oct 30 16:39:28.426437 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:28.426401    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:28.426527 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:28.426475    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:28.426527 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:28.426508    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:29.304115 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:29.304074    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" event={"ID":"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c","Type":"ContainerStarted","Data":"cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544"}
Oct 30 16:39:29.304631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:29.304127    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" event={"ID":"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c","Type":"ContainerStarted","Data":"44e22a4b094b63b5a1077fc39d51054f53836d400b89c343c6a286343179abc3"}
Oct 30 16:39:29.357970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:29.357914    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" podStartSLOduration=2.357899674 podStartE2EDuration="2.357899674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:29.35601668 +0000 UTC m=+999.209894945" watchObservedRunningTime="2026-10-30 16:39:29.357899674 +0000 UTC m=+999.211777939"
Oct 30 16:39:30.096463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:30.096406    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:30.098614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:30.098586    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 16:39:30.098753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:30.098642    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 16:39:31.096292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:31.093605    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 16:39:31.096292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:31.093683    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 16:39:31.301617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:31.301587    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:39:32.097220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:32.096559    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 16:39:32.097220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:32.096633    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 16:39:33.098205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:33.093275    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 16:39:33.098205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:33.093345    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 16:39:33.251885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:33.251858    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:39:34.096374 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:34.096336    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 16:39:34.096822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:34.096779    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 16:39:35.092939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:35.092901    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 16:39:35.093388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:35.092960    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 16:39:36.097675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.097646    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:36.098125 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.098073    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:36.098754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.098736    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 16:39:36.435956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.435882    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"]
Oct 30 16:39:36.440300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.440267    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.453226 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.453204    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-82381960-serving-cert\""
Oct 30 16:39:36.461298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.461270    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"]
Oct 30 16:39:36.533832 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.533801    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-openshift-service-ca\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.534015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.533858    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.534063 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.534019    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-metrics-certs\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.534156 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.534141    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49rq\" (UniqueName: \"kubernetes.io/projected/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-kube-api-access-z49rq\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.636275 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.636222    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49rq\" (UniqueName: \"kubernetes.io/projected/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-kube-api-access-z49rq\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.636497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.636370    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-openshift-service-ca\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.636497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.636430    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.636634 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.636573    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-metrics-certs\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.636963 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:39:36.636830    4497 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-82381960-serving-cert: secret "maas-api-e2e-mt-82381960-serving-cert" not found
Oct 30 16:39:36.636963 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:39:36.636916    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls podName:3f3f347a-10f3-4bbb-9f38-9e8ba54391ef nodeName:}" failed. No retries permitted until 2026-10-30 16:39:37.136895596 +0000 UTC m=+1006.990773851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls") pod "maas-api-e2e-mt-82381960-696b6bf757-pdvhd" (UID: "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef") : secret "maas-api-e2e-mt-82381960-serving-cert" not found
Oct 30 16:39:36.637832 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.637798    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-openshift-service-ca\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.648503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.640531    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-metrics-certs\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:36.678248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:36.658776    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49rq\" (UniqueName: \"kubernetes.io/projected/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-kube-api-access-z49rq\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:37.144417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:37.144385    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:37.146876 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:37.146850    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls\") pod \"maas-api-e2e-mt-82381960-696b6bf757-pdvhd\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:37.354963 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:37.354935    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:37.683692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:37.683592    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"]
Oct 30 16:39:37.686702 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:39:37.686677    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3f347a_10f3_4bbb_9f38_9e8ba54391ef.slice/crio-86dd556a40621f1574bf41f65c186e771ad2499e8c7ebeee76756ef6b1e114ae WatchSource:0}: Error finding container 86dd556a40621f1574bf41f65c186e771ad2499e8c7ebeee76756ef6b1e114ae: Status 404 returned error can't find the container with id 86dd556a40621f1574bf41f65c186e771ad2499e8c7ebeee76756ef6b1e114ae
Oct 30 16:39:38.375338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:38.375231    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd" event={"ID":"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef","Type":"ContainerStarted","Data":"1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697"}
Oct 30 16:39:38.375338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:38.375295    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd" event={"ID":"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef","Type":"ContainerStarted","Data":"86dd556a40621f1574bf41f65c186e771ad2499e8c7ebeee76756ef6b1e114ae"}
Oct 30 16:39:38.375785 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:38.375434    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:38.416292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:38.416093    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd" podStartSLOduration=2.078137773 podStartE2EDuration="2.416074492s" totalImagesPullingTime="337.936719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:38.412995761 +0000 UTC m=+1008.266874026" watchObservedRunningTime="2026-10-30 16:39:38.416074492 +0000 UTC m=+1008.269952767"
Oct 30 16:39:44.409260 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:44.409180    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:39:47.275401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.275360    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"]
Oct 30 16:39:47.280479 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.280459    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.306998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.306960    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"]
Oct 30 16:39:47.343577 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.343543    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-90df54-openshift-default-dockercfg-gv2k6\""
Oct 30 16:39:47.455823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.455761    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27051549-670c-4ee5-8a2f-4fee4f272067-istiod-ca-cert\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.455823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.455826    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27051549-670c-4ee5-8a2f-4fee4f272067-istio-podinfo\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.456081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.455970    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-istio-token\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.456146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.456121    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tg9k\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-kube-api-access-9tg9k\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.456226 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.456208    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-data\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.456330 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.456313    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.456482 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.456465    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-certs\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.456541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.456519    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-envoy\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.456597 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.456569    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-credential-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.558649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.558545    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tg9k\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-kube-api-access-9tg9k\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.558649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.558618    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-data\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.558889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.558701    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.558889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.558749    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-certs\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.558889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.558778    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-envoy\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.558889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.558813    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-credential-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.559109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.558934    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27051549-670c-4ee5-8a2f-4fee4f272067-istiod-ca-cert\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.559109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.558963    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27051549-670c-4ee5-8a2f-4fee4f272067-istio-podinfo\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.559109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.559054    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-istio-token\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.559819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.559790    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-data\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.559819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.559808    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-certs\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.560077 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.560058    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-credential-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.560276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.560253    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.560426 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.560404    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27051549-670c-4ee5-8a2f-4fee4f272067-istiod-ca-cert\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.563615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.563586    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27051549-670c-4ee5-8a2f-4fee4f272067-istio-podinfo\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.564624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.564596    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-envoy\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.580007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.579970    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-istio-token\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.580282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.580260    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tg9k\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-kube-api-access-9tg9k\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.600382 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.600335    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:47.988604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.988570    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"]
Oct 30 16:39:47.991471 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:39:47.991444    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27051549_670c_4ee5_8a2f_4fee4f272067.slice/crio-3d18cc3bf7602f909ed3dcc2591965ef72d1a31c1cf820801a6a44da600c0fd3 WatchSource:0}: Error finding container 3d18cc3bf7602f909ed3dcc2591965ef72d1a31c1cf820801a6a44da600c0fd3: Status 404 returned error can't find the container with id 3d18cc3bf7602f909ed3dcc2591965ef72d1a31c1cf820801a6a44da600c0fd3
Oct 30 16:39:47.994430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.994385    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:47.994516 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.994484    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:47.994554 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:47.994533    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:48.496604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.496559    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" event={"ID":"27051549-670c-4ee5-8a2f-4fee4f272067","Type":"ContainerStarted","Data":"b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39"}
Oct 30 16:39:48.497074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.496630    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" event={"ID":"27051549-670c-4ee5-8a2f-4fee4f272067","Type":"ContainerStarted","Data":"3d18cc3bf7602f909ed3dcc2591965ef72d1a31c1cf820801a6a44da600c0fd3"}
Oct 30 16:39:48.498284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.498175    4497 generic.go:356] "Generic (PLEG): container finished" podID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerID="c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6" exitCode=0
Oct 30 16:39:48.498284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.498253    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" event={"ID":"aa339320-3f28-4d4a-9331-179ff100cdb1","Type":"ContainerDied","Data":"c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6"}
Oct 30 16:39:48.498676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.498658    4497 scope.go:122] "RemoveContainer" containerID="c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6"
Oct 30 16:39:48.500161 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.500129    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:48.500272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.500212    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:48.500272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.500246    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:48.539326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.539277    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" podStartSLOduration=1.53926181 podStartE2EDuration="1.53926181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:48.538059114 +0000 UTC m=+1018.391937380" watchObservedRunningTime="2026-10-30 16:39:48.53926181 +0000 UTC m=+1018.393140074"
Oct 30 16:39:48.600964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:48.600879    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:49.506852 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:49.506818    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" event={"ID":"aa339320-3f28-4d4a-9331-179ff100cdb1","Type":"ContainerStarted","Data":"f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36"}
Oct 30 16:39:49.606305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:49.606275    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:50.363348 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:50.363309    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:50.364854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:50.364829    4497 patch_prober.go:29] interesting pod/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 16:39:50.364973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:50.364877    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 16:39:50.512839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:50.512807    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:50.513732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:50.513700    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:39:51.364479 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:51.364448    4497 patch_prober.go:29] interesting pod/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 16:39:51.364662 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:51.364501    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 16:39:51.927088 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:39:51.927054    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:39:52.364015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:52.363980    4497 patch_prober.go:29] interesting pod/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 16:39:52.364237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:52.364035    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 16:39:53.364480 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:53.364444    4497 patch_prober.go:29] interesting pod/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 16:39:53.365804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:53.365763    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 16:39:54.153698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:54.153652    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:54.364532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:54.364490    4497 patch_prober.go:29] interesting pod/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 16:39:54.364961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:54.364553    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 16:39:55.204657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.204615    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"]
Oct 30 16:39:55.216116 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.216085    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.234402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.234292    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"]
Oct 30 16:39:55.236454 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.236426    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-83d885-openshift-default-dockercfg-nf6p7\""
Oct 30 16:39:55.236855 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.236833    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-data\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.236967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.236873    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmk5w\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-kube-api-access-nmk5w\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.236967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.236941    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-credential-socket\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.238151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.237038    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-token\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.238151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.237160    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-certs\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.238151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.237210    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-envoy\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.238151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.237339    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-socket\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.238151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.237490    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0fc41e-ab04-42e2-8f1f-301485dada69-istiod-ca-cert\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.238151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.237561    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-podinfo\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.339298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339259    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-data\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.339515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339312    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmk5w\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-kube-api-access-nmk5w\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.339515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339381    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-credential-socket\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.339515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339479    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-token\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.339680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339607    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-certs\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.339680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339636    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-envoy\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.340201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339734    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-socket\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.340201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339765    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0fc41e-ab04-42e2-8f1f-301485dada69-istiod-ca-cert\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.340201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339791    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-podinfo\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.340201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.339831    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-data\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.342212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.340700    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-certs\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.342212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.340932    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-socket\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.342438 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.342252    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0fc41e-ab04-42e2-8f1f-301485dada69-istiod-ca-cert\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.342617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.342594    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-credential-socket\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.343558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.343097    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-podinfo\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.344784 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.344758    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-envoy\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.376263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.372946    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmk5w\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-kube-api-access-nmk5w\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.411211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.393593    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:55.424370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.418759    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-token\") pod \"e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") " pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:55.424370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.420708    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:39:55.546208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:55.545018    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:56.029253 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:56.029160    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"]
Oct 30 16:39:56.032548 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:39:56.032517    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0fc41e_ab04_42e2_8f1f_301485dada69.slice/crio-cddf7e8e76ceed785958d9506b134a37e385a2eed68b104f450bc6295b77d295 WatchSource:0}: Error finding container cddf7e8e76ceed785958d9506b134a37e385a2eed68b104f450bc6295b77d295: Status 404 returned error can't find the container with id cddf7e8e76ceed785958d9506b134a37e385a2eed68b104f450bc6295b77d295
Oct 30 16:39:56.035686 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:56.035648    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:56.035804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:56.035734    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:56.035864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:56.035804    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:39:56.667106 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:56.667069    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" event={"ID":"ad0fc41e-ab04-42e2-8f1f-301485dada69","Type":"ContainerStarted","Data":"67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e"}
Oct 30 16:39:56.667656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:56.667129    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" event={"ID":"ad0fc41e-ab04-42e2-8f1f-301485dada69","Type":"ContainerStarted","Data":"cddf7e8e76ceed785958d9506b134a37e385a2eed68b104f450bc6295b77d295"}
Oct 30 16:39:56.727646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:56.727592    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podStartSLOduration=1.7275786210000001 podStartE2EDuration="1.727578621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:56.726747929 +0000 UTC m=+1026.580626217" watchObservedRunningTime="2026-10-30 16:39:56.727578621 +0000 UTC m=+1026.581456886"
Oct 30 16:39:57.546946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:57.546906    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:57.552903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:57.552878    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:57.698212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:57.698037    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:39:57.699396 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:39:57.699374    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:40:02.746633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:02.745703    4497 patch_prober.go:29] interesting pod/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:53586->10.133.0.82:15021: read: connection reset by peer" start-of-body=
Oct 30 16:40:02.746633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:02.745811    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": read tcp 10.133.0.2:53586->10.133.0.82:15021: read: connection reset by peer"
Oct 30 16:40:03.761765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:03.761731    4497 generic.go:356] "Generic (PLEG): container finished" podID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerID="603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25" exitCode=0
Oct 30 16:40:03.762268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:03.761790    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" event={"ID":"a16e1d36-6550-473d-b77d-4cba06f6279d","Type":"ContainerDied","Data":"603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25"}
Oct 30 16:40:03.762268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:03.762241    4497 scope.go:122] "RemoveContainer" containerID="603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25"
Oct 30 16:40:03.763737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:03.763710    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:40:03.763809 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:03.763780    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:40:03.763845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:03.763821    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:40:04.789212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:04.786699    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" event={"ID":"a16e1d36-6550-473d-b77d-4cba06f6279d","Type":"ContainerStarted","Data":"8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51"}
Oct 30 16:40:05.493211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:05.492615    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:40:05.496327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:05.496299    4497 patch_prober.go:29] interesting pod/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 16:40:05.496450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:05.496352    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 16:40:06.496624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:06.496586    4497 patch_prober.go:29] interesting pod/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 16:40:06.497025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:06.496659    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 16:40:07.496623 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:07.496579    4497 patch_prober.go:29] interesting pod/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 16:40:07.496842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:07.496658    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 16:40:08.492380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:08.492344    4497 patch_prober.go:29] interesting pod/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 16:40:08.492679 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:08.492648    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 16:40:09.494754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:09.494718    4497 patch_prober.go:29] interesting pod/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 16:40:09.495386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:09.495350    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 16:40:10.416841 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.416811    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"]
Oct 30 16:40:10.421410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.421390    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.443053 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.443029    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-90df54-serving-cert\""
Oct 30 16:40:10.457221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.457175    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"]
Oct 30 16:40:10.496902 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.496874    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:40:10.497341 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.497228    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:40:10.498301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.498278    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:40:10.514907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.514882    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-maas-api-tls\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.515055 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.515014    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-metrics-certs\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.515156 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.515140    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87vh9\" (UniqueName: \"kubernetes.io/projected/084418cc-3a0f-4462-9e99-cd1831113cca-kube-api-access-87vh9\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.515266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.515254    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/084418cc-3a0f-4462-9e99-cd1831113cca-openshift-service-ca\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.617065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.617032    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-maas-api-tls\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.617300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.617136    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-metrics-certs\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.619054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.619025    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87vh9\" (UniqueName: \"kubernetes.io/projected/084418cc-3a0f-4462-9e99-cd1831113cca-kube-api-access-87vh9\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.620199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.619910    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/084418cc-3a0f-4462-9e99-cd1831113cca-openshift-service-ca\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.621382 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.620417    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-maas-api-tls\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.621485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.620976    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/084418cc-3a0f-4462-9e99-cd1831113cca-openshift-service-ca\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.626539 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.626515    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-metrics-certs\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.641020 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.640988    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87vh9\" (UniqueName: \"kubernetes.io/projected/084418cc-3a0f-4462-9e99-cd1831113cca-kube-api-access-87vh9\") pod \"maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:10.735516 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:10.735426    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:11.048038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:11.047966    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"]
Oct 30 16:40:11.051215 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:40:11.051170    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084418cc_3a0f_4462_9e99_cd1831113cca.slice/crio-4f029a7b928d3fd6e28e786b385e312224321fd8781483769ceac9ab8eb9f86b WatchSource:0}: Error finding container 4f029a7b928d3fd6e28e786b385e312224321fd8781483769ceac9ab8eb9f86b: Status 404 returned error can't find the container with id 4f029a7b928d3fd6e28e786b385e312224321fd8781483769ceac9ab8eb9f86b
Oct 30 16:40:11.866918 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:11.866874    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7" event={"ID":"084418cc-3a0f-4462-9e99-cd1831113cca","Type":"ContainerStarted","Data":"5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5"}
Oct 30 16:40:11.867451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:11.866946    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7" event={"ID":"084418cc-3a0f-4462-9e99-cd1831113cca","Type":"ContainerStarted","Data":"4f029a7b928d3fd6e28e786b385e312224321fd8781483769ceac9ab8eb9f86b"}
Oct 30 16:40:11.867451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:11.867003    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:11.907715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:11.907663    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7" podStartSLOduration=1.56673297 podStartE2EDuration="1.907643207s" totalImagesPullingTime="340.910237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:11.904866615 +0000 UTC m=+1041.758744879" watchObservedRunningTime="2026-10-30 16:40:11.907643207 +0000 UTC m=+1041.761521474"
Oct 30 16:40:17.881237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:17.881203    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:40:21.917721 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:40:21.917683    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:40:36.896601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:36.896537    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:40:37.073342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:37.073287    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:40:37.074079 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:37.073613    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded"
Oct 30 16:40:37.789381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:37.789340    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6v4bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:40:37.789729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:37.789697    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:40:37.916553 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:37.916503    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:40:37.996751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:37.996710    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:40:38.099274 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.073905    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Oct 30 16:40:38.441709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.226422    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"]
Oct 30 16:40:38.522769 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.246727    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 16:40:38.522769 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.399571    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:40:38.535453 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.535426    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:40:38.545137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.543966    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:38.557834 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.557639    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7410c7-dockercfg-96hgc\""
Oct 30 16:40:38.557834 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.557771    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7410c7\""
Oct 30 16:40:38.568169 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.568145    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"]
Oct 30 16:40:38.568476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.568458    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"]
Oct 30 16:40:38.574323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.574303    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:38.579756 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.578971    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:40:38.595813 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.595782    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ded80-f53a-439a-9ae8-316a0580343e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg\" (UID: \"e81ded80-f53a-439a-9ae8-316a0580343e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:38.596228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.596207    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czk45\" (UniqueName: \"kubernetes.io/projected/e81ded80-f53a-439a-9ae8-316a0580343e-kube-api-access-czk45\") pod \"payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg\" (UID: \"e81ded80-f53a-439a-9ae8-316a0580343e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:38.620818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.611603    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:40:38.640954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.640917    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"]
Oct 30 16:40:38.703068 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.702969    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306f358-0e4e-4b68-a776-c419d8a8b386-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt\" (UID: \"d306f358-0e4e-4b68-a776-c419d8a8b386\") " pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:38.703523 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.703496    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vvv\" (UniqueName: \"kubernetes.io/projected/d306f358-0e4e-4b68-a776-c419d8a8b386-kube-api-access-w7vvv\") pod \"payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt\" (UID: \"d306f358-0e4e-4b68-a776-c419d8a8b386\") " pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:38.703803 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.703785    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ded80-f53a-439a-9ae8-316a0580343e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg\" (UID: \"e81ded80-f53a-439a-9ae8-316a0580343e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:38.704061 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.704044    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czk45\" (UniqueName: \"kubernetes.io/projected/e81ded80-f53a-439a-9ae8-316a0580343e-kube-api-access-czk45\") pod \"payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg\" (UID: \"e81ded80-f53a-439a-9ae8-316a0580343e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:38.706382 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.706356    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ded80-f53a-439a-9ae8-316a0580343e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg\" (UID: \"e81ded80-f53a-439a-9ae8-316a0580343e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:38.743215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.742264    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czk45\" (UniqueName: \"kubernetes.io/projected/e81ded80-f53a-439a-9ae8-316a0580343e-kube-api-access-czk45\") pod \"payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg\" (UID: \"e81ded80-f53a-439a-9ae8-316a0580343e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:38.842080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.842017    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vvv\" (UniqueName: \"kubernetes.io/projected/d306f358-0e4e-4b68-a776-c419d8a8b386-kube-api-access-w7vvv\") pod \"payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt\" (UID: \"d306f358-0e4e-4b68-a776-c419d8a8b386\") " pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:38.842363 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.842343    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306f358-0e4e-4b68-a776-c419d8a8b386-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt\" (UID: \"d306f358-0e4e-4b68-a776-c419d8a8b386\") " pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:38.843384 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.843358    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306f358-0e4e-4b68-a776-c419d8a8b386-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt\" (UID: \"d306f358-0e4e-4b68-a776-c419d8a8b386\") " pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:38.867890 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.867858    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vvv\" (UniqueName: \"kubernetes.io/projected/d306f358-0e4e-4b68-a776-c419d8a8b386-kube-api-access-w7vvv\") pod \"payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt\" (UID: \"d306f358-0e4e-4b68-a776-c419d8a8b386\") " pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:38.879208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.877468    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:38.943305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:38.943266    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:39.257593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.257564    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"]
Oct 30 16:40:39.260597 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:40:39.260570    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81ded80_f53a_439a_9ae8_316a0580343e.slice/crio-13c230c78a4af0256831ab146856b314df0358d4a95e03d3e6fdb7a24e1988ed WatchSource:0}: Error finding container 13c230c78a4af0256831ab146856b314df0358d4a95e03d3e6fdb7a24e1988ed: Status 404 returned error can't find the container with id 13c230c78a4af0256831ab146856b314df0358d4a95e03d3e6fdb7a24e1988ed
Oct 30 16:40:39.305213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.305175    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"]
Oct 30 16:40:39.307139 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:40:39.307116    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd306f358_0e4e_4b68_a776_c419d8a8b386.slice/crio-9d87640ec0fb3a832c7d6b058fc4ecc5572be0f5790130457a4777fdbb903d6c WatchSource:0}: Error finding container 9d87640ec0fb3a832c7d6b058fc4ecc5572be0f5790130457a4777fdbb903d6c: Status 404 returned error can't find the container with id 9d87640ec0fb3a832c7d6b058fc4ecc5572be0f5790130457a4777fdbb903d6c
Oct 30 16:40:39.437629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.437571    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt" event={"ID":"d306f358-0e4e-4b68-a776-c419d8a8b386","Type":"ContainerStarted","Data":"560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d"}
Oct 30 16:40:39.438098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.437646    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt" event={"ID":"d306f358-0e4e-4b68-a776-c419d8a8b386","Type":"ContainerStarted","Data":"9d87640ec0fb3a832c7d6b058fc4ecc5572be0f5790130457a4777fdbb903d6c"}
Oct 30 16:40:39.438098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.437807    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:39.439497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.439471    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg" event={"ID":"e81ded80-f53a-439a-9ae8-316a0580343e","Type":"ContainerStarted","Data":"1cf8ff60e573d9a8957680e3c644975ce2903138e95530fa7919d21ff6052e25"}
Oct 30 16:40:39.439635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.439523    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg" event={"ID":"e81ded80-f53a-439a-9ae8-316a0580343e","Type":"ContainerStarted","Data":"13c230c78a4af0256831ab146856b314df0358d4a95e03d3e6fdb7a24e1988ed"}
Oct 30 16:40:39.439635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.439569    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:39.502509 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.502452    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt" podStartSLOduration=1.502436145 podStartE2EDuration="1.502436145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:39.497656205 +0000 UTC m=+1069.351534471" watchObservedRunningTime="2026-10-30 16:40:39.502436145 +0000 UTC m=+1069.356314410"
Oct 30 16:40:39.536528 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:39.536412    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg" podStartSLOduration=1.5363964220000002 podStartE2EDuration="1.536396422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:39.533166979 +0000 UTC m=+1069.387045256" watchObservedRunningTime="2026-10-30 16:40:39.536396422 +0000 UTC m=+1069.390274686"
Oct 30 16:40:43.739313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:43.739282    4497 patch_prober.go:29] interesting pod/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:40:43.739699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:43.739360    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:40:46.487214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:46.487091    4497 generic.go:356] "Generic (PLEG): container finished" podID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerID="67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e" exitCode=0
Oct 30 16:40:46.487214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:46.487178    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" event={"ID":"ad0fc41e-ab04-42e2-8f1f-301485dada69","Type":"ContainerDied","Data":"67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e"}
Oct 30 16:40:46.487731 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:46.487714    4497 scope.go:122] "RemoveContainer" containerID="67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e"
Oct 30 16:40:46.489719 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:46.489682    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:40:46.489887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:46.489765    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:40:46.489887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:46.489821    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:40:47.505207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:47.505140    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" event={"ID":"ad0fc41e-ab04-42e2-8f1f-301485dada69","Type":"ContainerStarted","Data":"a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07"}
Oct 30 16:40:47.546826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:47.546782    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:40:47.547061 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:47.547036    4497 patch_prober.go:29] interesting pod/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 16:40:47.547148 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:47.547097    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 16:40:48.547022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:48.546986    4497 patch_prober.go:29] interesting pod/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 16:40:48.547590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:48.547045    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 16:40:49.546999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:49.546965    4497 patch_prober.go:29] interesting pod/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 16:40:49.547173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:49.547018    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 16:40:50.454088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:50.452831    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:40:50.454476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:50.454454    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:40:50.547404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:50.547359    4497 patch_prober.go:29] interesting pod/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 16:40:50.549003 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:50.547434    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 16:40:51.547386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:51.547351    4497 patch_prober.go:29] interesting pod/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 16:40:51.547568 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:51.547405    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 16:40:51.927400 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:40:51.927293    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:40:52.547414 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:52.547375    4497 patch_prober.go:29] interesting pod/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 16:40:52.547622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:52.547463    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 16:40:53.551063 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:53.551037    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:40:53.551494 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:53.551423    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:40:53.552013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:53.551997    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:40:56.451802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:40:56.451766    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"]
Oct 30 16:41:01.452983 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.452910    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq" podUID="60b23937-07d3-468f-a78b-987629d92056" containerName="payload-pre-processing" containerID="cri-o://69da91f9771b7359e14b04f4f2b3417c089e8a2164e73d23399b190514be854a" gracePeriod=25
Oct 30 16:41:01.608127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.608095    4497 generic.go:356] "Generic (PLEG): container finished" podID="60b23937-07d3-468f-a78b-987629d92056" containerID="69da91f9771b7359e14b04f4f2b3417c089e8a2164e73d23399b190514be854a" exitCode=0
Oct 30 16:41:01.608324 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.608180    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq" event={"ID":"60b23937-07d3-468f-a78b-987629d92056","Type":"ContainerDied","Data":"69da91f9771b7359e14b04f4f2b3417c089e8a2164e73d23399b190514be854a"}
Oct 30 16:41:01.728792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.728763    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:41:01.897929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.897887    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b23937-07d3-468f-a78b-987629d92056-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b23937-07d3-468f-a78b-987629d92056-plugins-config-volume\") pod \"60b23937-07d3-468f-a78b-987629d92056\" (UID: \"60b23937-07d3-468f-a78b-987629d92056\") "
Oct 30 16:41:01.898080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.897990    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b23937-07d3-468f-a78b-987629d92056-kube-api-access-p76ws\" (UniqueName: \"kubernetes.io/projected/60b23937-07d3-468f-a78b-987629d92056-kube-api-access-p76ws\") pod \"60b23937-07d3-468f-a78b-987629d92056\" (UID: \"60b23937-07d3-468f-a78b-987629d92056\") "
Oct 30 16:41:01.898344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.898322    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b23937-07d3-468f-a78b-987629d92056-plugins-config-volume" pod "60b23937-07d3-468f-a78b-987629d92056" (UID: "60b23937-07d3-468f-a78b-987629d92056"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:41:01.898515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.898501    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b23937-07d3-468f-a78b-987629d92056-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:01.900153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:01.900131    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b23937-07d3-468f-a78b-987629d92056-kube-api-access-p76ws" pod "60b23937-07d3-468f-a78b-987629d92056" (UID: "60b23937-07d3-468f-a78b-987629d92056"). InnerVolumeSpecName "kube-api-access-p76ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:41:02.000448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:02.000357    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p76ws\" (UniqueName: \"kubernetes.io/projected/60b23937-07d3-468f-a78b-987629d92056-kube-api-access-p76ws\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:02.618353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:02.618262    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"
Oct 30 16:41:02.618353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:02.618296    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq" event={"ID":"60b23937-07d3-468f-a78b-987629d92056","Type":"ContainerDied","Data":"4dd8eccb3a2318e3c8d7ab42881dfe2502c383bb7ce999a43f9f117d3e1c9506"}
Oct 30 16:41:02.618887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:02.618370    4497 scope.go:122] "RemoveContainer" containerID="69da91f9771b7359e14b04f4f2b3417c089e8a2164e73d23399b190514be854a"
Oct 30 16:41:02.663734 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:02.663706    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"]
Oct 30 16:41:02.672656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:02.672624    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0b09f1-8b8746c79-s5htq"]
Oct 30 16:41:04.300952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:04.300917    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b23937-07d3-468f-a78b-987629d92056" path="/var/lib/kubelet/pods/60b23937-07d3-468f-a78b-987629d92056/volumes"
Oct 30 16:41:21.917861 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:41:21.917824    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:41:22.319003 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.318950    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"]
Oct 30 16:41:22.319990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.319963    4497 cpu_manager.go:414] "containerMap: removing container" podUID="60b23937-07d3-468f-a78b-987629d92056" containerName="payload-pre-processing"
Oct 30 16:41:22.320123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.319992    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b23937-07d3-468f-a78b-987629d92056" containerName="payload-pre-processing"
Oct 30 16:41:22.320254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.320237    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b23937-07d3-468f-a78b-987629d92056" containerName="payload-pre-processing"
Oct 30 16:41:22.323499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.323472    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.339898 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.339858    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1189d4-openshift-default-dockercfg-s6fsb\""
Oct 30 16:41:22.356365 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.356334    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"]
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.530753    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2w6j\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-kube-api-access-k2w6j\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.530815    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-certs\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.530846    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-data\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.530931    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-socket\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.530970    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-istio-token\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.531067    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6738852d-9876-451d-8b86-9718d8eb47c4-istiod-ca-cert\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.531125    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-credential-socket\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.531226    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-envoy\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.532216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.531365    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6738852d-9876-451d-8b86-9718d8eb47c4-istio-podinfo\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.632705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.632596    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6738852d-9876-451d-8b86-9718d8eb47c4-istio-podinfo\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.632883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.632729    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2w6j\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-kube-api-access-k2w6j\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.632883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.632753    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-certs\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.632883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.632778    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-data\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.632883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.632835    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-socket\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.632883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.632855    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-istio-token\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.633052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.632907    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6738852d-9876-451d-8b86-9718d8eb47c4-istiod-ca-cert\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.633052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.632945    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-credential-socket\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.633052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.633003    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-envoy\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.633658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.633627    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-certs\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.633912 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.633891    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-data\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.634001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.633973    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-socket\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.634167 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.634114    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-credential-socket\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.635048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.635015    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6738852d-9876-451d-8b86-9718d8eb47c4-istiod-ca-cert\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.638354 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.638330    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-envoy\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.638508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.638482    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6738852d-9876-451d-8b86-9718d8eb47c4-istio-podinfo\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.654762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.654725    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-istio-token\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.654912 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.654842    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2w6j\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-kube-api-access-k2w6j\") pod \"e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") " pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:22.941972 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:22.941868    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:23.273002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:23.272971    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"]
Oct 30 16:41:23.275374 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:41:23.275345    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6738852d_9876_451d_8b86_9718d8eb47c4.slice/crio-107fd2fa642979f1649f2f9d6bde1465515c183d2c0f0d1d7ff980081ffc280b WatchSource:0}: Error finding container 107fd2fa642979f1649f2f9d6bde1465515c183d2c0f0d1d7ff980081ffc280b: Status 404 returned error can't find the container with id 107fd2fa642979f1649f2f9d6bde1465515c183d2c0f0d1d7ff980081ffc280b
Oct 30 16:41:23.278126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:23.278091    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:41:23.278238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:23.278162    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:41:23.278238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:23.278207    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:41:23.806505 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:23.806458    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" event={"ID":"6738852d-9876-451d-8b86-9718d8eb47c4","Type":"ContainerStarted","Data":"de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e"}
Oct 30 16:41:23.806716 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:23.806570    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" event={"ID":"6738852d-9876-451d-8b86-9718d8eb47c4","Type":"ContainerStarted","Data":"107fd2fa642979f1649f2f9d6bde1465515c183d2c0f0d1d7ff980081ffc280b"}
Oct 30 16:41:23.868484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:23.868428    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" podStartSLOduration=1.868410172 podStartE2EDuration="1.868410172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:23.857296425 +0000 UTC m=+1113.711174689" watchObservedRunningTime="2026-10-30 16:41:23.868410172 +0000 UTC m=+1113.722288436"
Oct 30 16:41:23.942427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:23.942382    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:24.948558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:24.948527    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:25.819805 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:25.819769    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:25.820892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:25.820871    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:41:28.402207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.401524    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"]
Oct 30 16:41:28.452213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.406003    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"]
Oct 30 16:41:28.452213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.406124    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:28.478210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.453884    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1189d4-dockercfg-n7pzv\""
Oct 30 16:41:28.484737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.457875    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1189d4\""
Oct 30 16:41:28.607577 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.607534    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpj25\" (UniqueName: \"kubernetes.io/projected/b41536cd-79f3-4672-b457-e909c708b5bc-kube-api-access-tpj25\") pod \"payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24\" (UID: \"b41536cd-79f3-4672-b457-e909c708b5bc\") " pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:28.607838 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.607682    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41536cd-79f3-4672-b457-e909c708b5bc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24\" (UID: \"b41536cd-79f3-4672-b457-e909c708b5bc\") " pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:28.709358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.709264    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41536cd-79f3-4672-b457-e909c708b5bc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24\" (UID: \"b41536cd-79f3-4672-b457-e909c708b5bc\") " pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:28.709564 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.709548    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpj25\" (UniqueName: \"kubernetes.io/projected/b41536cd-79f3-4672-b457-e909c708b5bc-kube-api-access-tpj25\") pod \"payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24\" (UID: \"b41536cd-79f3-4672-b457-e909c708b5bc\") " pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:28.710044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.710021    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41536cd-79f3-4672-b457-e909c708b5bc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24\" (UID: \"b41536cd-79f3-4672-b457-e909c708b5bc\") " pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:28.730815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.730778    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpj25\" (UniqueName: \"kubernetes.io/projected/b41536cd-79f3-4672-b457-e909c708b5bc-kube-api-access-tpj25\") pod \"payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24\" (UID: \"b41536cd-79f3-4672-b457-e909c708b5bc\") " pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:28.808708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:28.808674    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:29.116838 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:29.116812    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"]
Oct 30 16:41:29.120182 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:41:29.120150    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41536cd_79f3_4672_b457_e909c708b5bc.slice/crio-d4dcac71d1fc26c75c78813bee4d9f49ea0ea2a08cc2659e15b66ecc683bc32b WatchSource:0}: Error finding container d4dcac71d1fc26c75c78813bee4d9f49ea0ea2a08cc2659e15b66ecc683bc32b: Status 404 returned error can't find the container with id d4dcac71d1fc26c75c78813bee4d9f49ea0ea2a08cc2659e15b66ecc683bc32b
Oct 30 16:41:29.852355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:29.852320    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24" event={"ID":"b41536cd-79f3-4672-b457-e909c708b5bc","Type":"ContainerStarted","Data":"6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54"}
Oct 30 16:41:29.852824 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:29.852373    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24" event={"ID":"b41536cd-79f3-4672-b457-e909c708b5bc","Type":"ContainerStarted","Data":"d4dcac71d1fc26c75c78813bee4d9f49ea0ea2a08cc2659e15b66ecc683bc32b"}
Oct 30 16:41:29.852824 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:29.852394    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:29.890106 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:29.890046    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24" podStartSLOduration=1.890031022 podStartE2EDuration="1.890031022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:29.888903515 +0000 UTC m=+1119.742781782" watchObservedRunningTime="2026-10-30 16:41:29.890031022 +0000 UTC m=+1119.743909286"
Oct 30 16:41:32.945443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:32.945363    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"]
Oct 30 16:41:32.945889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:32.945560    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5" podUID="46ae9315-d893-4c58-b16b-e8cc6faf3045" containerName="maas-api" containerID="cri-o://398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d" gracePeriod=30
Oct 30 16:41:33.321055 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.321013    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:41:33.510211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.509635    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-maas-api-tls\") pod \"46ae9315-d893-4c58-b16b-e8cc6faf3045\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") "
Oct 30 16:41:33.510211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.509747    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ae9315-d893-4c58-b16b-e8cc6faf3045-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae9315-d893-4c58-b16b-e8cc6faf3045-openshift-service-ca\") pod \"46ae9315-d893-4c58-b16b-e8cc6faf3045\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") "
Oct 30 16:41:33.510211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.510122    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ae9315-d893-4c58-b16b-e8cc6faf3045-openshift-service-ca" pod "46ae9315-d893-4c58-b16b-e8cc6faf3045" (UID: "46ae9315-d893-4c58-b16b-e8cc6faf3045"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:41:33.510519 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.510221    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-metrics-certs\") pod \"46ae9315-d893-4c58-b16b-e8cc6faf3045\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") "
Oct 30 16:41:33.510519 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.510270    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ae9315-d893-4c58-b16b-e8cc6faf3045-kube-api-access-8ldqf\" (UniqueName: \"kubernetes.io/projected/46ae9315-d893-4c58-b16b-e8cc6faf3045-kube-api-access-8ldqf\") pod \"46ae9315-d893-4c58-b16b-e8cc6faf3045\" (UID: \"46ae9315-d893-4c58-b16b-e8cc6faf3045\") "
Oct 30 16:41:33.514570 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.511022    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae9315-d893-4c58-b16b-e8cc6faf3045-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:33.514570 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.512087    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-maas-api-tls" pod "46ae9315-d893-4c58-b16b-e8cc6faf3045" (UID: "46ae9315-d893-4c58-b16b-e8cc6faf3045"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:41:33.520053 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.514948    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-metrics-certs" pod "46ae9315-d893-4c58-b16b-e8cc6faf3045" (UID: "46ae9315-d893-4c58-b16b-e8cc6faf3045"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:41:33.564211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.557395    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ae9315-d893-4c58-b16b-e8cc6faf3045-kube-api-access-8ldqf" pod "46ae9315-d893-4c58-b16b-e8cc6faf3045" (UID: "46ae9315-d893-4c58-b16b-e8cc6faf3045"). InnerVolumeSpecName "kube-api-access-8ldqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:41:33.624339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.612708    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:33.624339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.612750    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldqf\" (UniqueName: \"kubernetes.io/projected/46ae9315-d893-4c58-b16b-e8cc6faf3045-kube-api-access-8ldqf\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:33.624339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.612766    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ae9315-d893-4c58-b16b-e8cc6faf3045-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:33.880842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.880762    4497 generic.go:356] "Generic (PLEG): container finished" podID="46ae9315-d893-4c58-b16b-e8cc6faf3045" containerID="398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d" exitCode=0
Oct 30 16:41:33.880842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.880803    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5" event={"ID":"46ae9315-d893-4c58-b16b-e8cc6faf3045","Type":"ContainerDied","Data":"398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d"}
Oct 30 16:41:33.880842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.880813    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"
Oct 30 16:41:33.881102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.880844    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5" event={"ID":"46ae9315-d893-4c58-b16b-e8cc6faf3045","Type":"ContainerDied","Data":"4f60e6009c9ee7056337883c9e3eeab0ca62837d0d0515d574db415ab419ea6f"}
Oct 30 16:41:33.881102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.880865    4497 scope.go:122] "RemoveContainer" containerID="398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d"
Oct 30 16:41:33.891065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.891036    4497 scope.go:122] "RemoveContainer" containerID="398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d"
Oct 30 16:41:33.891415 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:41:33.891396    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d\": container with ID starting with 398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d not found: ID does not exist" containerID="398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d"
Oct 30 16:41:33.891495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.891425    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d"} err="failed to get container status \"398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d\": rpc error: code = NotFound desc = could not find container \"398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d\": container with ID starting with 398e3894685ea235f87e060703057b641c09d6a8126da73585e91d4de2dcae2d not found: ID does not exist"
Oct 30 16:41:33.925602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.925570    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"]
Oct 30 16:41:33.934982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:33.934954    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8295b2-7fdd869899-t7ll5"]
Oct 30 16:41:34.305079 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:34.305038    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ae9315-d893-4c58-b16b-e8cc6faf3045" path="/var/lib/kubelet/pods/46ae9315-d893-4c58-b16b-e8cc6faf3045/volumes"
Oct 30 16:41:40.862598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:40.862564    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:41:41.524753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:41.524718    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"]
Oct 30 16:41:41.524980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:41.524956    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy" containerID="cri-o://f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36" gracePeriod=30
Oct 30 16:41:44.648911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:44.648857    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"]
Oct 30 16:41:44.649606 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:44.649131    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy" containerID="cri-o://8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51" gracePeriod=30
Oct 30 16:41:46.822304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.822276    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:41:46.973752 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.973717    4497 generic.go:356] "Generic (PLEG): container finished" podID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerID="f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36" exitCode=0
Oct 30 16:41:46.973935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.973763    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"
Oct 30 16:41:46.973935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.973779    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" event={"ID":"aa339320-3f28-4d4a-9331-179ff100cdb1","Type":"ContainerDied","Data":"f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36"}
Oct 30 16:41:46.973935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.973838    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd" event={"ID":"aa339320-3f28-4d4a-9331-179ff100cdb1","Type":"ContainerDied","Data":"7045545f2ea631adaacaa0f5c3e88bd105a32a4854326df109a9dbbdea900318"}
Oct 30 16:41:46.973935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.973867    4497 scope.go:122] "RemoveContainer" containerID="f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36"
Oct 30 16:41:46.984041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.984021    4497 scope.go:122] "RemoveContainer" containerID="c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6"
Oct 30 16:41:46.989744 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.989722    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-token\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-token\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.989848 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.989777    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-kube-api-access-9w22w\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-kube-api-access-9w22w\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.989848 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.989816    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-credential-socket\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.989934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.989871    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-socket\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.990022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990006    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-data\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.990072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990041    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-envoy\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.990196 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990116    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-credential-socket" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:46.990268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990199    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-socket" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:46.990268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990245    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa339320-3f28-4d4a-9331-179ff100cdb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa339320-3f28-4d4a-9331-179ff100cdb1-istiod-ca-cert\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.990380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990344    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-certs\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.990380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990362    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-data" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:46.990497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990389    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-podinfo\") pod \"aa339320-3f28-4d4a-9331-179ff100cdb1\" (UID: \"aa339320-3f28-4d4a-9331-179ff100cdb1\") "
Oct 30 16:41:46.990646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990622    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-certs" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:46.990735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.990649    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa339320-3f28-4d4a-9331-179ff100cdb1-istiod-ca-cert" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:41:46.991151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.991129    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:46.991245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.991158    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:46.991245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.991172    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:46.991245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.991220    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa339320-3f28-4d4a-9331-179ff100cdb1-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:46.991245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.991235    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:46.992127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.992104    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-token" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:41:46.992451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.992430    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-kube-api-access-9w22w" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "kube-api-access-9w22w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:41:46.992630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.992610    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-envoy" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:46.992630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.992596    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-podinfo" pod "aa339320-3f28-4d4a-9331-179ff100cdb1" (UID: "aa339320-3f28-4d4a-9331-179ff100cdb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:41:46.994762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.994744    4497 scope.go:122] "RemoveContainer" containerID="f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36"
Oct 30 16:41:46.995081 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:41:46.995062    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36\": container with ID starting with f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36 not found: ID does not exist" containerID="f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36"
Oct 30 16:41:46.995152 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.995088    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36"} err="failed to get container status \"f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36\": rpc error: code = NotFound desc = could not find container \"f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36\": container with ID starting with f2c09271391cad7066691c2189ea05e74925b6ab592277cda8edbaf6e7440f36 not found: ID does not exist"
Oct 30 16:41:46.995152 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.995106    4497 scope.go:122] "RemoveContainer" containerID="c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6"
Oct 30 16:41:46.995406 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:41:46.995384    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6\": container with ID starting with c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6 not found: ID does not exist" containerID="c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6"
Oct 30 16:41:46.995460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:46.995415    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6"} err="failed to get container status \"c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6\": rpc error: code = NotFound desc = could not find container \"c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6\": container with ID starting with c31659ae4e3592beb0c444c006967456727ce7953259a74805db8d209f48d6b6 not found: ID does not exist"
Oct 30 16:41:47.092399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:47.092305    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:47.092399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:47.092333    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:47.092399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:47.092346    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:47.092399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:47.092356    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w22w\" (UniqueName: \"kubernetes.io/projected/aa339320-3f28-4d4a-9331-179ff100cdb1-kube-api-access-9w22w\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:47.320781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:47.320748    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"]
Oct 30 16:41:47.329091 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:47.329060    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8295b2-openshift-default-6b5464f449-7m9rd"]
Oct 30 16:41:48.300941 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:48.300904    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" path="/var/lib/kubelet/pods/aa339320-3f28-4d4a-9331-179ff100cdb1/volumes"
Oct 30 16:41:49.949057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:49.949031    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:41:49.996522 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:49.996493    4497 generic.go:356] "Generic (PLEG): container finished" podID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerID="8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51" exitCode=0
Oct 30 16:41:49.996705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:49.996540    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"
Oct 30 16:41:49.996705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:49.996573    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" event={"ID":"a16e1d36-6550-473d-b77d-4cba06f6279d","Type":"ContainerDied","Data":"8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51"}
Oct 30 16:41:49.996705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:49.996637    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx" event={"ID":"a16e1d36-6550-473d-b77d-4cba06f6279d","Type":"ContainerDied","Data":"5ba75d26d0eee2f4ed1c7720b5af6708c259bfe5eb3065d08970bdf1f4eab97e"}
Oct 30 16:41:49.996705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:49.996667    4497 scope.go:122] "RemoveContainer" containerID="8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51"
Oct 30 16:41:50.007718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.007692    4497 scope.go:122] "RemoveContainer" containerID="603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25"
Oct 30 16:41:50.019233 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.019164    4497 scope.go:122] "RemoveContainer" containerID="8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51"
Oct 30 16:41:50.019567 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:41:50.019543    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51\": container with ID starting with 8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51 not found: ID does not exist" containerID="8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51"
Oct 30 16:41:50.019661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.019573    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51"} err="failed to get container status \"8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51\": rpc error: code = NotFound desc = could not find container \"8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51\": container with ID starting with 8778cd0e11d34f4ad4642fb6589f4226a530744c312adbf6b2d165bc78af8a51 not found: ID does not exist"
Oct 30 16:41:50.019661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.019591    4497 scope.go:122] "RemoveContainer" containerID="603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25"
Oct 30 16:41:50.019882 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:41:50.019857    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25\": container with ID starting with 603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25 not found: ID does not exist" containerID="603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25"
Oct 30 16:41:50.019987 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.019892    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25"} err="failed to get container status \"603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25\": rpc error: code = NotFound desc = could not find container \"603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25\": container with ID starting with 603e36be2bee43992532c696479ad3ad36ea5c23e520dbf9a5b827b77fe1bd25 not found: ID does not exist"
Oct 30 16:41:50.036200 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036155    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16e1d36-6550-473d-b77d-4cba06f6279d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16e1d36-6550-473d-b77d-4cba06f6279d-istiod-ca-cert\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036314 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036217    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-podinfo\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036314 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036253    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-certs\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036371    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-token\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-token\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036436    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-data\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036550    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-kube-api-access-w5ht6\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-kube-api-access-w5ht6\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036584    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-socket\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036594    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16e1d36-6550-473d-b77d-4cba06f6279d-istiod-ca-cert" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:41:50.036635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036607    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-certs" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:50.036635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036612    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-envoy\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036836 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036701    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-credential-socket\") pod \"a16e1d36-6550-473d-b77d-4cba06f6279d\" (UID: \"a16e1d36-6550-473d-b77d-4cba06f6279d\") "
Oct 30 16:41:50.036890 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036869    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-socket" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:50.037004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.036979    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-data" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:50.037070 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.037047    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-credential-socket" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:50.037576 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.037548    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.037693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.037579    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.037693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.037596    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16e1d36-6550-473d-b77d-4cba06f6279d-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.037693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.037610    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.037693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.037623    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.039044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.039021    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-podinfo" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:41:50.040495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.040457    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-envoy" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:41:50.040660 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.040638    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-kube-api-access-w5ht6" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "kube-api-access-w5ht6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:41:50.042644 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.042612    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-token" pod "a16e1d36-6550-473d-b77d-4cba06f6279d" (UID: "a16e1d36-6550-473d-b77d-4cba06f6279d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:41:50.138563 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.138529    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.138563 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.138560    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5ht6\" (UniqueName: \"kubernetes.io/projected/a16e1d36-6550-473d-b77d-4cba06f6279d-kube-api-access-w5ht6\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.138563 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.138569    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.138868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.138578    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16e1d36-6550-473d-b77d-4cba06f6279d-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:50.343869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.343839    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"]
Oct 30 16:41:50.352623 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:50.352594    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0b09f1-openshift-default-546dbfbfcc-scpbx"]
Oct 30 16:41:51.917608 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:41:51.917571    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:41:52.301341 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:52.301306    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" path="/var/lib/kubelet/pods/a16e1d36-6550-473d-b77d-4cba06f6279d/volumes"
Oct 30 16:41:58.404402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:58.404367    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"]
Oct 30 16:41:58.405103 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:58.404616    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd" podUID="3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" containerName="maas-api" containerID="cri-o://1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697" gracePeriod=30
Oct 30 16:41:58.822659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:58.820797    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:41:59.034790 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.034746    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-openshift-service-ca\") pod \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") "
Oct 30 16:41:59.035096 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.035077    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-kube-api-access-z49rq\" (UniqueName: \"kubernetes.io/projected/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-kube-api-access-z49rq\") pod \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") "
Oct 30 16:41:59.035247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.035215    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-metrics-certs\") pod \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") "
Oct 30 16:41:59.035343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.035256    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls\") pod \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\" (UID: \"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef\") "
Oct 30 16:41:59.047347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.042011    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-openshift-service-ca" pod "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" (UID: "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:41:59.053864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.053782    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls" pod "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" (UID: "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:41:59.054036 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.053951    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-kube-api-access-z49rq" pod "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" (UID: "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef"). InnerVolumeSpecName "kube-api-access-z49rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:41:59.054779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.054550    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-metrics-certs" pod "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" (UID: "3f3f347a-10f3-4bbb-9f38-9e8ba54391ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:41:59.077246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.076539    4497 generic.go:356] "Generic (PLEG): container finished" podID="3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" containerID="1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697" exitCode=0
Oct 30 16:41:59.077246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.076618    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"
Oct 30 16:41:59.077246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.076642    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd" event={"ID":"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef","Type":"ContainerDied","Data":"1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697"}
Oct 30 16:41:59.077246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.076718    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd" event={"ID":"3f3f347a-10f3-4bbb-9f38-9e8ba54391ef","Type":"ContainerDied","Data":"86dd556a40621f1574bf41f65c186e771ad2499e8c7ebeee76756ef6b1e114ae"}
Oct 30 16:41:59.077246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.076753    4497 scope.go:122] "RemoveContainer" containerID="1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697"
Oct 30 16:41:59.096458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.096432    4497 scope.go:122] "RemoveContainer" containerID="1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697"
Oct 30 16:41:59.096905 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:41:59.096876    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697\": container with ID starting with 1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697 not found: ID does not exist" containerID="1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697"
Oct 30 16:41:59.096987 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.096914    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697"} err="failed to get container status \"1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697\": rpc error: code = NotFound desc = could not find container \"1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697\": container with ID starting with 1c0ae07b6a4c3bb90f961fbd5d390b069c31b5c141db73fdcb45f97d45ede697 not found: ID does not exist"
Oct 30 16:41:59.123992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.123957    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"]
Oct 30 16:41:59.127649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.127621    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-pdvhd"]
Oct 30 16:41:59.138002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.137973    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z49rq\" (UniqueName: \"kubernetes.io/projected/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-kube-api-access-z49rq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:59.138138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.138007    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:59.138138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.138024    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:41:59.138138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:41:59.138038    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:00.301486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:00.301454    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" path="/var/lib/kubelet/pods/3f3f347a-10f3-4bbb-9f38-9e8ba54391ef/volumes"
Oct 30 16:42:03.109393 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:03.109302    4497 generic.go:356] "Generic (PLEG): container finished" podID="27051549-670c-4ee5-8a2f-4fee4f272067" containerID="b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39" exitCode=0
Oct 30 16:42:03.109837 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:03.109375    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" event={"ID":"27051549-670c-4ee5-8a2f-4fee4f272067","Type":"ContainerDied","Data":"b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39"}
Oct 30 16:42:03.109886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:03.109845    4497 scope.go:122] "RemoveContainer" containerID="b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39"
Oct 30 16:42:03.111448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:03.111418    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:42:03.111569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:03.111505    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:42:03.111569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:03.111558    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:42:04.119486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:04.119454    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" event={"ID":"27051549-670c-4ee5-8a2f-4fee4f272067","Type":"ContainerStarted","Data":"06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559"}
Oct 30 16:42:04.602942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:04.602466    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:42:04.602942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:04.602812    4497 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 16:42:04.602942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:04.602866    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 16:42:05.521209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:05.513623    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:42:05.601675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:05.601635    4497 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 16:42:05.601856 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:05.601689    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 16:42:06.619209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:06.618347    4497 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 16:42:06.619209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:06.618435    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 16:42:07.619019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:07.618978    4497 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 16:42:07.619341 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:07.619309    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 16:42:08.613212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:08.604988    4497 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 16:42:08.613212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:08.605061    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 16:42:09.732237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:09.732200    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:42:09.734050 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:09.734024    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:42:11.890415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.890382    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"]
Oct 30 16:42:11.891041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891027    4497 cpu_manager.go:414] "containerMap: removing container" podUID="46ae9315-d893-4c58-b16b-e8cc6faf3045" containerName="maas-api"
Oct 30 16:42:11.891084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891042    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ae9315-d893-4c58-b16b-e8cc6faf3045" containerName="maas-api"
Oct 30 16:42:11.891084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891062    4497 cpu_manager.go:414] "containerMap: removing container" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy"
Oct 30 16:42:11.891084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891067    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy"
Oct 30 16:42:11.891182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891128    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" containerName="maas-api"
Oct 30 16:42:11.891182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891137    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" containerName="maas-api"
Oct 30 16:42:11.891182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891151    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy"
Oct 30 16:42:11.891182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891159    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy"
Oct 30 16:42:11.891352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891195    4497 cpu_manager.go:414] "containerMap: removing container" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy"
Oct 30 16:42:11.891352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891205    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy"
Oct 30 16:42:11.891352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891246    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy"
Oct 30 16:42:11.891352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891252    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy"
Oct 30 16:42:11.891483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891407    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3f347a-10f3-4bbb-9f38-9e8ba54391ef" containerName="maas-api"
Oct 30 16:42:11.891483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891418    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy"
Oct 30 16:42:11.891483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891424    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy"
Oct 30 16:42:11.891483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891434    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16e1d36-6550-473d-b77d-4cba06f6279d" containerName="istio-proxy"
Oct 30 16:42:11.891483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.891445    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ae9315-d893-4c58-b16b-e8cc6faf3045" containerName="maas-api"
Oct 30 16:42:11.894089 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.894071    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:11.915718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.915678    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-82381960-dockercfg-dzwql\""
Oct 30 16:42:11.915718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.915695    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-82381960\""
Oct 30 16:42:11.921305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.921275    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"]
Oct 30 16:42:11.931785 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.931752    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khbqd\" (UniqueName: \"kubernetes.io/projected/768da413-799e-48c3-ae2c-c3bd651d018a-kube-api-access-khbqd\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg\" (UID: \"768da413-799e-48c3-ae2c-c3bd651d018a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:11.931931 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:11.931840    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768da413-799e-48c3-ae2c-c3bd651d018a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg\" (UID: \"768da413-799e-48c3-ae2c-c3bd651d018a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:12.065903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.065861    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khbqd\" (UniqueName: \"kubernetes.io/projected/768da413-799e-48c3-ae2c-c3bd651d018a-kube-api-access-khbqd\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg\" (UID: \"768da413-799e-48c3-ae2c-c3bd651d018a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:12.066219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.066180    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768da413-799e-48c3-ae2c-c3bd651d018a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg\" (UID: \"768da413-799e-48c3-ae2c-c3bd651d018a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:12.076635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.076592    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768da413-799e-48c3-ae2c-c3bd651d018a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg\" (UID: \"768da413-799e-48c3-ae2c-c3bd651d018a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:12.137604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.134442    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"]
Oct 30 16:42:12.137604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.135657    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa339320-3f28-4d4a-9331-179ff100cdb1" containerName="istio-proxy"
Oct 30 16:42:12.142980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.142920    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"]
Oct 30 16:42:12.143296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.143237    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:12.151541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.151498    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khbqd\" (UniqueName: \"kubernetes.io/projected/768da413-799e-48c3-ae2c-c3bd651d018a-kube-api-access-khbqd\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg\" (UID: \"768da413-799e-48c3-ae2c-c3bd651d018a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:12.208030 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.208001    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:12.269327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.269287    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qrn\" (UniqueName: \"kubernetes.io/projected/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-kube-api-access-x6qrn\") pod \"payload-processing-e2e-mt-82381960-579cc97966-hxbh5\" (UID: \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:12.269534 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.269365    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-82381960-579cc97966-hxbh5\" (UID: \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:12.370658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.370624    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qrn\" (UniqueName: \"kubernetes.io/projected/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-kube-api-access-x6qrn\") pod \"payload-processing-e2e-mt-82381960-579cc97966-hxbh5\" (UID: \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:12.370833 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.370712    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-82381960-579cc97966-hxbh5\" (UID: \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:12.371694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.371667    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-plugins-config-volume\") pod \"payload-processing-e2e-mt-82381960-579cc97966-hxbh5\" (UID: \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:12.392105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.392070    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qrn\" (UniqueName: \"kubernetes.io/projected/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-kube-api-access-x6qrn\") pod \"payload-processing-e2e-mt-82381960-579cc97966-hxbh5\" (UID: \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:12.490146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.490067    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:12.530906 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.530832    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"]
Oct 30 16:42:12.810008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:12.809669    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"]
Oct 30 16:42:12.814518 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:42:12.814490    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa39be64_f7d1_4a1c_9e27_548948f1a0b6.slice/crio-d488ba3615b8f1d09ac2baaf4a66780c0b2ad987df2cdd9872e59e2ffe0cd184 WatchSource:0}: Error finding container d488ba3615b8f1d09ac2baaf4a66780c0b2ad987df2cdd9872e59e2ffe0cd184: Status 404 returned error can't find the container with id d488ba3615b8f1d09ac2baaf4a66780c0b2ad987df2cdd9872e59e2ffe0cd184
Oct 30 16:42:13.196765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:13.196672    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg" event={"ID":"768da413-799e-48c3-ae2c-c3bd651d018a","Type":"ContainerStarted","Data":"f15f21a5f5cc64947aad4a6b80a8f04196d469e900de630988152f8fd54c30e9"}
Oct 30 16:42:13.196765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:13.196725    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg" event={"ID":"768da413-799e-48c3-ae2c-c3bd651d018a","Type":"ContainerStarted","Data":"1bfcf77911be58952ac1925b89170c7d6b9f363f264aae6d61d6237b52b4635b"}
Oct 30 16:42:13.197355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:13.196775    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:13.198631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:13.198608    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5" event={"ID":"aa39be64-f7d1-4a1c-9e27-548948f1a0b6","Type":"ContainerStarted","Data":"ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1"}
Oct 30 16:42:13.198764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:13.198648    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5" event={"ID":"aa39be64-f7d1-4a1c-9e27-548948f1a0b6","Type":"ContainerStarted","Data":"d488ba3615b8f1d09ac2baaf4a66780c0b2ad987df2cdd9872e59e2ffe0cd184"}
Oct 30 16:42:13.198764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:13.198699    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:13.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:13.233471    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg" podStartSLOduration=2.233455785 podStartE2EDuration="2.233455785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:13.231828493 +0000 UTC m=+1163.085706760" watchObservedRunningTime="2026-10-30 16:42:13.233455785 +0000 UTC m=+1163.087334049"
Oct 30 16:42:13.293605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:13.293550    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5" podStartSLOduration=2.293534736 podStartE2EDuration="2.293534736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:13.291377858 +0000 UTC m=+1163.145256119" watchObservedRunningTime="2026-10-30 16:42:13.293534736 +0000 UTC m=+1163.147413027"
Oct 30 16:42:15.216684 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:15.216583    4497 generic.go:356] "Generic (PLEG): container finished" podID="6738852d-9876-451d-8b86-9718d8eb47c4" containerID="de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e" exitCode=0
Oct 30 16:42:15.216684 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:15.216656    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" event={"ID":"6738852d-9876-451d-8b86-9718d8eb47c4","Type":"ContainerDied","Data":"de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e"}
Oct 30 16:42:15.217123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:15.217056    4497 scope.go:122] "RemoveContainer" containerID="de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e"
Oct 30 16:42:15.218440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:15.218413    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:42:15.218498 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:15.218481    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:42:15.218532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:15.218516    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:42:16.111646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.111610    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"]
Oct 30 16:42:16.115375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.115351    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.127616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.127588    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-82381960-serving-cert\""
Oct 30 16:42:16.148656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.148618    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"]
Oct 30 16:42:16.225011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.224970    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" event={"ID":"6738852d-9876-451d-8b86-9718d8eb47c4","Type":"ContainerStarted","Data":"b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948"}
Oct 30 16:42:16.233033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.233000    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7f113-46fc-4efd-b9ad-729351d065d9-openshift-service-ca\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.233173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.233130    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-maas-api-tls\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.233305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.233289    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfwl\" (UniqueName: \"kubernetes.io/projected/4de7f113-46fc-4efd-b9ad-729351d065d9-kube-api-access-pjfwl\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.233410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.233397    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-metrics-certs\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.334449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.334413    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-metrics-certs\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.334645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.334493    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7f113-46fc-4efd-b9ad-729351d065d9-openshift-service-ca\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.334645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.334563    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-maas-api-tls\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.334772 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.334735    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfwl\" (UniqueName: \"kubernetes.io/projected/4de7f113-46fc-4efd-b9ad-729351d065d9-kube-api-access-pjfwl\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.335330 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.335305    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7f113-46fc-4efd-b9ad-729351d065d9-openshift-service-ca\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.337131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.337106    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-metrics-certs\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.337261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.337229    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-maas-api-tls\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.355777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.355752    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfwl\" (UniqueName: \"kubernetes.io/projected/4de7f113-46fc-4efd-b9ad-729351d065d9-kube-api-access-pjfwl\") pod \"maas-api-e2e-mt-82381960-696b6bf757-bntsx\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.428891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.428777    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:16.799391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.799358    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"]
Oct 30 16:42:16.801421 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:42:16.801395    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de7f113_46fc_4efd_b9ad_729351d065d9.slice/crio-ed078164958c9024e07441eae1d8a478318615e52d100fb36958d6221b674013 WatchSource:0}: Error finding container ed078164958c9024e07441eae1d8a478318615e52d100fb36958d6221b674013: Status 404 returned error can't find the container with id ed078164958c9024e07441eae1d8a478318615e52d100fb36958d6221b674013
Oct 30 16:42:16.942155 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.942117    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:42:16.942492 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.942469    4497 patch_prober.go:29] interesting pod/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 16:42:16.942540 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:16.942517    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 16:42:17.233321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:17.233286    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx" event={"ID":"4de7f113-46fc-4efd-b9ad-729351d065d9","Type":"ContainerStarted","Data":"ed078164958c9024e07441eae1d8a478318615e52d100fb36958d6221b674013"}
Oct 30 16:42:17.942939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:17.942894    4497 patch_prober.go:29] interesting pod/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 16:42:17.943141 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:17.942965    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 16:42:18.246266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.245232    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx" event={"ID":"4de7f113-46fc-4efd-b9ad-729351d065d9","Type":"ContainerStarted","Data":"9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca"}
Oct 30 16:42:18.251177 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.246736    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:18.328286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.327129    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx" podStartSLOduration=1.96200614 podStartE2EDuration="2.327113222s" totalImagesPullingTime="365.107082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:18.325368936 +0000 UTC m=+1168.179247201" watchObservedRunningTime="2026-10-30 16:42:18.327113222 +0000 UTC m=+1168.180991487"
Oct 30 16:42:18.459207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.458389    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"]
Oct 30 16:42:18.459207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.458626    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7" podUID="084418cc-3a0f-4462-9e99-cd1831113cca" containerName="maas-api" containerID="cri-o://5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5" gracePeriod=30
Oct 30 16:42:18.942892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.942501    4497 patch_prober.go:29] interesting pod/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 16:42:18.942892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.942561    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 16:42:18.943244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.943209    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"]
Oct 30 16:42:18.954688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:18.954660    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:42:19.005214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.004310    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"]
Oct 30 16:42:19.058124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.058088    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-maas-api-tls\") pod \"084418cc-3a0f-4462-9e99-cd1831113cca\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") "
Oct 30 16:42:19.059849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.059820    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-metrics-certs\") pod \"084418cc-3a0f-4462-9e99-cd1831113cca\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") "
Oct 30 16:42:19.060084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.060060    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-maas-api-tls" pod "084418cc-3a0f-4462-9e99-cd1831113cca" (UID: "084418cc-3a0f-4462-9e99-cd1831113cca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:42:19.060215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.060116    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084418cc-3a0f-4462-9e99-cd1831113cca-kube-api-access-87vh9\" (UniqueName: \"kubernetes.io/projected/084418cc-3a0f-4462-9e99-cd1831113cca-kube-api-access-87vh9\") pod \"084418cc-3a0f-4462-9e99-cd1831113cca\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") "
Oct 30 16:42:19.060417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.060400    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084418cc-3a0f-4462-9e99-cd1831113cca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/084418cc-3a0f-4462-9e99-cd1831113cca-openshift-service-ca\") pod \"084418cc-3a0f-4462-9e99-cd1831113cca\" (UID: \"084418cc-3a0f-4462-9e99-cd1831113cca\") "
Oct 30 16:42:19.061389 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.061371    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:19.061691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.061664    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084418cc-3a0f-4462-9e99-cd1831113cca-openshift-service-ca" pod "084418cc-3a0f-4462-9e99-cd1831113cca" (UID: "084418cc-3a0f-4462-9e99-cd1831113cca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:42:19.062345 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.062326    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084418cc-3a0f-4462-9e99-cd1831113cca-kube-api-access-87vh9" pod "084418cc-3a0f-4462-9e99-cd1831113cca" (UID: "084418cc-3a0f-4462-9e99-cd1831113cca"). InnerVolumeSpecName "kube-api-access-87vh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:42:19.097221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.094323    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-metrics-certs" pod "084418cc-3a0f-4462-9e99-cd1831113cca" (UID: "084418cc-3a0f-4462-9e99-cd1831113cca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:42:19.162865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.162833    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/084418cc-3a0f-4462-9e99-cd1831113cca-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:19.162865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.162863    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87vh9\" (UniqueName: \"kubernetes.io/projected/084418cc-3a0f-4462-9e99-cd1831113cca-kube-api-access-87vh9\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:19.162865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.162872    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/084418cc-3a0f-4462-9e99-cd1831113cca-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:19.264998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.264959    4497 generic.go:356] "Generic (PLEG): container finished" podID="084418cc-3a0f-4462-9e99-cd1831113cca" containerID="5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5" exitCode=0
Oct 30 16:42:19.266510 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.266301    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7" event={"ID":"084418cc-3a0f-4462-9e99-cd1831113cca","Type":"ContainerDied","Data":"5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5"}
Oct 30 16:42:19.266510 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.266359    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7" event={"ID":"084418cc-3a0f-4462-9e99-cd1831113cca","Type":"ContainerDied","Data":"4f029a7b928d3fd6e28e786b385e312224321fd8781483769ceac9ab8eb9f86b"}
Oct 30 16:42:19.266510 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.266389    4497 scope.go:122] "RemoveContainer" containerID="5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5"
Oct 30 16:42:19.266822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.266807    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"
Oct 30 16:42:19.349558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.349528    4497 scope.go:122] "RemoveContainer" containerID="5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5"
Oct 30 16:42:19.360962 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:19.360923    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5\": container with ID starting with 5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5 not found: ID does not exist" containerID="5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5"
Oct 30 16:42:19.361142 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.360973    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5"} err="failed to get container status \"5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5\": rpc error: code = NotFound desc = could not find container \"5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5\": container with ID starting with 5f21d6ab91c139ec1ddca2301da097752a36357a84f2cdf8fa9999a57147c2e5 not found: ID does not exist"
Oct 30 16:42:19.440691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.440659    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"]
Oct 30 16:42:19.445510 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.445470    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-90df54-79fb47c5d-lcbl7"]
Oct 30 16:42:19.943000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.942968    4497 patch_prober.go:29] interesting pod/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 16:42:19.943217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:19.943043    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 16:42:20.305407 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:20.305374    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084418cc-3a0f-4462-9e99-cd1831113cca" path="/var/lib/kubelet/pods/084418cc-3a0f-4462-9e99-cd1831113cca/volumes"
Oct 30 16:42:20.942546 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:20.942510    4497 patch_prober.go:29] interesting pod/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 16:42:20.942708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:20.942594    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 16:42:21.926110 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:21.926068    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:42:21.946712 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:21.946687    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:42:21.947093 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:21.947075    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:42:21.947819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:21.947801    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:42:23.943615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:23.943568    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt" podUID="d306f358-0e4e-4b68-a776-c419d8a8b386" containerName="payload-processing" containerID="cri-o://560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d" gracePeriod=25
Oct 30 16:42:24.005374 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.005328    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg" podUID="e81ded80-f53a-439a-9ae8-316a0580343e" containerName="payload-pre-processing" containerID="cri-o://1cf8ff60e573d9a8957680e3c644975ce2903138e95530fa7919d21ff6052e25" gracePeriod=25
Oct 30 16:42:24.220329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.220298    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:42:24.222182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.222159    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:42:24.283701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.283670    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:42:24.328926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.328890    4497 generic.go:356] "Generic (PLEG): container finished" podID="d306f358-0e4e-4b68-a776-c419d8a8b386" containerID="560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d" exitCode=0
Oct 30 16:42:24.329212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.329084    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt" event={"ID":"d306f358-0e4e-4b68-a776-c419d8a8b386","Type":"ContainerDied","Data":"560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d"}
Oct 30 16:42:24.329212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.329135    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt" event={"ID":"d306f358-0e4e-4b68-a776-c419d8a8b386","Type":"ContainerDied","Data":"9d87640ec0fb3a832c7d6b058fc4ecc5572be0f5790130457a4777fdbb903d6c"}
Oct 30 16:42:24.329212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.329165    4497 scope.go:122] "RemoveContainer" containerID="560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d"
Oct 30 16:42:24.329451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.329347    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"
Oct 30 16:42:24.337893 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.337860    4497 generic.go:356] "Generic (PLEG): container finished" podID="e81ded80-f53a-439a-9ae8-316a0580343e" containerID="1cf8ff60e573d9a8957680e3c644975ce2903138e95530fa7919d21ff6052e25" exitCode=0
Oct 30 16:42:24.338018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.337914    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg" event={"ID":"e81ded80-f53a-439a-9ae8-316a0580343e","Type":"ContainerDied","Data":"1cf8ff60e573d9a8957680e3c644975ce2903138e95530fa7919d21ff6052e25"}
Oct 30 16:42:24.338018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.337961    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg" event={"ID":"e81ded80-f53a-439a-9ae8-316a0580343e","Type":"ContainerDied","Data":"13c230c78a4af0256831ab146856b314df0358d4a95e03d3e6fdb7a24e1988ed"}
Oct 30 16:42:24.338018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.337985    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13c230c78a4af0256831ab146856b314df0358d4a95e03d3e6fdb7a24e1988ed"
Oct 30 16:42:24.342829 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.342806    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:42:24.349055 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.349033    4497 scope.go:122] "RemoveContainer" containerID="560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d"
Oct 30 16:42:24.349384 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:24.349362    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d\": container with ID starting with 560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d not found: ID does not exist" containerID="560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d"
Oct 30 16:42:24.349467 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.349396    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d"} err="failed to get container status \"560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d\": rpc error: code = NotFound desc = could not find container \"560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d\": container with ID starting with 560fc156eb2ba3fa032a8281432e646099d5e624e94f11f53e0a284ea7652f6d not found: ID does not exist"
Oct 30 16:42:24.355230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.354353    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81ded80-f53a-439a-9ae8-316a0580343e-kube-api-access-czk45\" (UniqueName: \"kubernetes.io/projected/e81ded80-f53a-439a-9ae8-316a0580343e-kube-api-access-czk45\") pod \"e81ded80-f53a-439a-9ae8-316a0580343e\" (UID: \"e81ded80-f53a-439a-9ae8-316a0580343e\") "
Oct 30 16:42:24.355230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.354452    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d306f358-0e4e-4b68-a776-c419d8a8b386-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306f358-0e4e-4b68-a776-c419d8a8b386-plugins-config-volume\") pod \"d306f358-0e4e-4b68-a776-c419d8a8b386\" (UID: \"d306f358-0e4e-4b68-a776-c419d8a8b386\") "
Oct 30 16:42:24.355230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.354647    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81ded80-f53a-439a-9ae8-316a0580343e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ded80-f53a-439a-9ae8-316a0580343e-plugins-config-volume\") pod \"e81ded80-f53a-439a-9ae8-316a0580343e\" (UID: \"e81ded80-f53a-439a-9ae8-316a0580343e\") "
Oct 30 16:42:24.355230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.354705    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d306f358-0e4e-4b68-a776-c419d8a8b386-kube-api-access-w7vvv\" (UniqueName: \"kubernetes.io/projected/d306f358-0e4e-4b68-a776-c419d8a8b386-kube-api-access-w7vvv\") pod \"d306f358-0e4e-4b68-a776-c419d8a8b386\" (UID: \"d306f358-0e4e-4b68-a776-c419d8a8b386\") "
Oct 30 16:42:24.355230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.355082    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d306f358-0e4e-4b68-a776-c419d8a8b386-plugins-config-volume" pod "d306f358-0e4e-4b68-a776-c419d8a8b386" (UID: "d306f358-0e4e-4b68-a776-c419d8a8b386"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:42:24.355230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.355104    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81ded80-f53a-439a-9ae8-316a0580343e-plugins-config-volume" pod "e81ded80-f53a-439a-9ae8-316a0580343e" (UID: "e81ded80-f53a-439a-9ae8-316a0580343e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:42:24.356793 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.356774    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81ded80-f53a-439a-9ae8-316a0580343e-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:24.356870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.356796    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d306f358-0e4e-4b68-a776-c419d8a8b386-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:24.363577 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.363550    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81ded80-f53a-439a-9ae8-316a0580343e-kube-api-access-czk45" pod "e81ded80-f53a-439a-9ae8-316a0580343e" (UID: "e81ded80-f53a-439a-9ae8-316a0580343e"). InnerVolumeSpecName "kube-api-access-czk45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:42:24.364465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.364438    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d306f358-0e4e-4b68-a776-c419d8a8b386-kube-api-access-w7vvv" pod "d306f358-0e4e-4b68-a776-c419d8a8b386" (UID: "d306f358-0e4e-4b68-a776-c419d8a8b386"). InnerVolumeSpecName "kube-api-access-w7vvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:42:24.457924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.457842    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vvv\" (UniqueName: \"kubernetes.io/projected/d306f358-0e4e-4b68-a776-c419d8a8b386-kube-api-access-w7vvv\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:24.457924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.457871    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czk45\" (UniqueName: \"kubernetes.io/projected/e81ded80-f53a-439a-9ae8-316a0580343e-kube-api-access-czk45\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:24.672563 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.672524    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"]
Oct 30 16:42:24.681444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:24.681417    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7410c7-88fbc4f78-6sgqt"]
Oct 30 16:42:25.307202 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:25.307162    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:42:25.345896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:25.345864    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"
Oct 30 16:42:25.398918 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:25.398882    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"]
Oct 30 16:42:25.404533 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:25.404504    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7410c7-f6b758f87-q6hwg"]
Oct 30 16:42:26.310566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:26.310526    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d306f358-0e4e-4b68-a776-c419d8a8b386" path="/var/lib/kubelet/pods/d306f358-0e4e-4b68-a776-c419d8a8b386/volumes"
Oct 30 16:42:26.310974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:26.310959    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81ded80-f53a-439a-9ae8-316a0580343e" path="/var/lib/kubelet/pods/e81ded80-f53a-439a-9ae8-316a0580343e/volumes"
Oct 30 16:42:26.925468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:26.925434    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"]
Oct 30 16:42:26.925748 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:26.925724    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy" containerID="cri-o://a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07" gracePeriod=30
Oct 30 16:42:28.021847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:28.021807    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"]
Oct 30 16:42:28.022346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:28.022073    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy" containerID="cri-o://06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559" gracePeriod=30
Oct 30 16:42:32.244665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.244637    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:42:32.392598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.392556    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-credential-socket\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.392792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.392663    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0fc41e-ab04-42e2-8f1f-301485dada69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0fc41e-ab04-42e2-8f1f-301485dada69-istiod-ca-cert\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.392792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.392781    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-podinfo\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.392917 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.392809    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-certs\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.392917 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.392809    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-credential-socket" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:32.392917 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.392890    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-token\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-token\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.393088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.392932    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-data\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.393088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.392997    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-envoy\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.393088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393024    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0fc41e-ab04-42e2-8f1f-301485dada69-istiod-ca-cert" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:42:32.393270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393108    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-certs" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:32.393270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393102    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-socket\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.393270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393238    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-data" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:32.393270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393249    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-kube-api-access-nmk5w\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-kube-api-access-nmk5w\") pod \"ad0fc41e-ab04-42e2-8f1f-301485dada69\" (UID: \"ad0fc41e-ab04-42e2-8f1f-301485dada69\") "
Oct 30 16:42:32.393481 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393317    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-socket" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:32.393928 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393910    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.393928 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393929    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0fc41e-ab04-42e2-8f1f-301485dada69-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.394081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393939    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.394081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393947    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.394081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.393954    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.395070 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395047    4497 generic.go:356] "Generic (PLEG): container finished" podID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerID="a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07" exitCode=0
Oct 30 16:42:32.395256 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395112    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"
Oct 30 16:42:32.395256 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395126    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" event={"ID":"ad0fc41e-ab04-42e2-8f1f-301485dada69","Type":"ContainerDied","Data":"a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07"}
Oct 30 16:42:32.395256 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395178    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg" event={"ID":"ad0fc41e-ab04-42e2-8f1f-301485dada69","Type":"ContainerDied","Data":"cddf7e8e76ceed785958d9506b134a37e385a2eed68b104f450bc6295b77d295"}
Oct 30 16:42:32.395432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395240    4497 scope.go:122] "RemoveContainer" containerID="a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07"
Oct 30 16:42:32.395621 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395588    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-podinfo" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:42:32.395707 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395675    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-envoy" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:32.395889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395872    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-kube-api-access-nmk5w" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "kube-api-access-nmk5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:42:32.395954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.395911    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-token" pod "ad0fc41e-ab04-42e2-8f1f-301485dada69" (UID: "ad0fc41e-ab04-42e2-8f1f-301485dada69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:42:32.425211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.425178    4497 scope.go:122] "RemoveContainer" containerID="67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e"
Oct 30 16:42:32.434642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.434622    4497 scope.go:122] "RemoveContainer" containerID="a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07"
Oct 30 16:42:32.434946 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:32.434915    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07\": container with ID starting with a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07 not found: ID does not exist" containerID="a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07"
Oct 30 16:42:32.435018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.434954    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07"} err="failed to get container status \"a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07\": rpc error: code = NotFound desc = could not find container \"a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07\": container with ID starting with a7fe886f63130c49a3e31e76d97ff785110d20740a4ea139d25fa1bef9a89e07 not found: ID does not exist"
Oct 30 16:42:32.435018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.434973    4497 scope.go:122] "RemoveContainer" containerID="67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e"
Oct 30 16:42:32.435248 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:32.435226    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e\": container with ID starting with 67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e not found: ID does not exist" containerID="67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e"
Oct 30 16:42:32.435349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.435252    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e"} err="failed to get container status \"67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e\": rpc error: code = NotFound desc = could not find container \"67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e\": container with ID starting with 67d5d733f4fc14ba956da2ab1d8d15a59f6abb0e1fc07e7235fbe5017dae4e3e not found: ID does not exist"
Oct 30 16:42:32.495922 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.495846    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.495922 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.495873    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.495922 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.495883    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmk5w\" (UniqueName: \"kubernetes.io/projected/ad0fc41e-ab04-42e2-8f1f-301485dada69-kube-api-access-nmk5w\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.495922 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.495891    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0fc41e-ab04-42e2-8f1f-301485dada69-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:32.738458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.738426    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"]
Oct 30 16:42:32.747584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:32.747514    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-83d885-openshift-default-594cdc7d55-jvscg"]
Oct 30 16:42:33.304086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.304012    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:42:33.405729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.405689    4497 generic.go:356] "Generic (PLEG): container finished" podID="27051549-670c-4ee5-8a2f-4fee4f272067" containerID="06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559" exitCode=0
Oct 30 16:42:33.405907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.405743    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"
Oct 30 16:42:33.405907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.405768    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" event={"ID":"27051549-670c-4ee5-8a2f-4fee4f272067","Type":"ContainerDied","Data":"06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559"}
Oct 30 16:42:33.405907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.405834    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7" event={"ID":"27051549-670c-4ee5-8a2f-4fee4f272067","Type":"ContainerDied","Data":"3d18cc3bf7602f909ed3dcc2591965ef72d1a31c1cf820801a6a44da600c0fd3"}
Oct 30 16:42:33.405907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.405865    4497 scope.go:122] "RemoveContainer" containerID="06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559"
Oct 30 16:42:33.412409 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.412382    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-envoy\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.412551 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.412443    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-kube-api-access-9tg9k\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-kube-api-access-9tg9k\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.412616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.412569    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-certs\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.412616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.412603    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-istio-token\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-istio-token\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.412727 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.412655    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27051549-670c-4ee5-8a2f-4fee4f272067-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27051549-670c-4ee5-8a2f-4fee4f272067-istio-podinfo\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.412727 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.412701    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-credential-socket\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.412895 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.412869    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-certs" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:33.413024 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.413002    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-credential-socket" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:33.413105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.413052    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-data\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.413172 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.413109    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-socket\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.413349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.413328    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27051549-670c-4ee5-8a2f-4fee4f272067-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27051549-670c-4ee5-8a2f-4fee4f272067-istiod-ca-cert\") pod \"27051549-670c-4ee5-8a2f-4fee4f272067\" (UID: \"27051549-670c-4ee5-8a2f-4fee4f272067\") "
Oct 30 16:42:33.413417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.413351    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-data" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:33.413581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.413560    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-socket" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:33.414016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.413862    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27051549-670c-4ee5-8a2f-4fee4f272067-istiod-ca-cert" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:42:33.414137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.414098    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.414137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.414120    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.414137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.414135    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27051549-670c-4ee5-8a2f-4fee4f272067-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.414352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.414150    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.414352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.414162    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.415002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.414982    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-kube-api-access-9tg9k" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "kube-api-access-9tg9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:42:33.415092 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.415010    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-istio-token" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:42:33.415135 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.415097    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27051549-670c-4ee5-8a2f-4fee4f272067-istio-podinfo" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:42:33.415169 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.415156    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-envoy" pod "27051549-670c-4ee5-8a2f-4fee4f272067" (UID: "27051549-670c-4ee5-8a2f-4fee4f272067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:42:33.417002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.416983    4497 scope.go:122] "RemoveContainer" containerID="b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39"
Oct 30 16:42:33.444346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.444327    4497 scope.go:122] "RemoveContainer" containerID="06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559"
Oct 30 16:42:33.444664 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:33.444640    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559\": container with ID starting with 06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559 not found: ID does not exist" containerID="06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559"
Oct 30 16:42:33.444729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.444674    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559"} err="failed to get container status \"06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559\": rpc error: code = NotFound desc = could not find container \"06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559\": container with ID starting with 06844f1a097eb39d6439ae25b7739e25c7d8b3738ab9c0057c05ae4cfe357559 not found: ID does not exist"
Oct 30 16:42:33.444729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.444692    4497 scope.go:122] "RemoveContainer" containerID="b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39"
Oct 30 16:42:33.444964 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:33.444940    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39\": container with ID starting with b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39 not found: ID does not exist" containerID="b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39"
Oct 30 16:42:33.445015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.444971    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39"} err="failed to get container status \"b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39\": rpc error: code = NotFound desc = could not find container \"b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39\": container with ID starting with b3be0983f2c9733b2a520e202000b516dd43ef0801d5b668026eaa2254516e39 not found: ID does not exist"
Oct 30 16:42:33.516158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.516118    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27051549-670c-4ee5-8a2f-4fee4f272067-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.516158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.516152    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27051549-670c-4ee5-8a2f-4fee4f272067-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.516158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.516165    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tg9k\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-kube-api-access-9tg9k\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.516433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.516178    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27051549-670c-4ee5-8a2f-4fee4f272067-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:33.748038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.748000    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"]
Oct 30 16:42:33.757220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:33.757169    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-w7bq7"]
Oct 30 16:42:34.301566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:34.301534    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" path="/var/lib/kubelet/pods/27051549-670c-4ee5-8a2f-4fee4f272067/volumes"
Oct 30 16:42:34.302081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:34.302066    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" path="/var/lib/kubelet/pods/ad0fc41e-ab04-42e2-8f1f-301485dada69/volumes"
Oct 30 16:42:43.869947 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.869910    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7440-458937"]
Oct 30 16:42:43.870826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870804    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e81ded80-f53a-439a-9ae8-316a0580343e" containerName="payload-pre-processing"
Oct 30 16:42:43.870939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870827    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81ded80-f53a-439a-9ae8-316a0580343e" containerName="payload-pre-processing"
Oct 30 16:42:43.870939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870865    4497 cpu_manager.go:414] "containerMap: removing container" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy"
Oct 30 16:42:43.870939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870874    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy"
Oct 30 16:42:43.870939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870890    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy"
Oct 30 16:42:43.870939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870897    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy"
Oct 30 16:42:43.870939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870915    4497 cpu_manager.go:414] "containerMap: removing container" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy"
Oct 30 16:42:43.870939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870922    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870956    4497 cpu_manager.go:414] "containerMap: removing container" podUID="084418cc-3a0f-4462-9e99-cd1831113cca" containerName="maas-api"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870964    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084418cc-3a0f-4462-9e99-cd1831113cca" containerName="maas-api"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870984    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d306f358-0e4e-4b68-a776-c419d8a8b386" containerName="payload-processing"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.870991    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d306f358-0e4e-4b68-a776-c419d8a8b386" containerName="payload-processing"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.871148    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="084418cc-3a0f-4462-9e99-cd1831113cca" containerName="maas-api"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.871158    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.871170    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d306f358-0e4e-4b68-a776-c419d8a8b386" containerName="payload-processing"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.871199    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81ded80-f53a-439a-9ae8-316a0580343e" containerName="payload-pre-processing"
Oct 30 16:42:43.871306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.871214    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy"
Oct 30 16:42:43.873845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.873823    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:43.886444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.886424    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qn5rq\""
Oct 30 16:42:43.892037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.892013    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7440-458937"]
Oct 30 16:42:43.976334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:43.976296    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fc9m\" (UniqueName: \"kubernetes.io/projected/e5aeb44f-aedb-4ed9-9eed-ae9764381724-kube-api-access-8fc9m\") pod \"test-curl-7440-458937\" (UID: \"e5aeb44f-aedb-4ed9-9eed-ae9764381724\") " pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:44.078216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:44.078150    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fc9m\" (UniqueName: \"kubernetes.io/projected/e5aeb44f-aedb-4ed9-9eed-ae9764381724-kube-api-access-8fc9m\") pod \"test-curl-7440-458937\" (UID: \"e5aeb44f-aedb-4ed9-9eed-ae9764381724\") " pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:44.099004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:44.098973    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fc9m\" (UniqueName: \"kubernetes.io/projected/e5aeb44f-aedb-4ed9-9eed-ae9764381724-kube-api-access-8fc9m\") pod \"test-curl-7440-458937\" (UID: \"e5aeb44f-aedb-4ed9-9eed-ae9764381724\") " pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:44.186396 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:44.186305    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:44.490935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:44.490689    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7440-458937"]
Oct 30 16:42:44.496525 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:42:44.496492    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5aeb44f_aedb_4ed9_9eed_ae9764381724.slice/crio-d5497739bdd1463fab10f54d4716e05251ece6b1a393b207e9f5ee274d133f92 WatchSource:0}: Error finding container d5497739bdd1463fab10f54d4716e05251ece6b1a393b207e9f5ee274d133f92: Status 404 returned error can't find the container with id d5497739bdd1463fab10f54d4716e05251ece6b1a393b207e9f5ee274d133f92
Oct 30 16:42:44.513277 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:44.513247    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7440-458937" event={"ID":"e5aeb44f-aedb-4ed9-9eed-ae9764381724","Type":"ContainerStarted","Data":"d5497739bdd1463fab10f54d4716e05251ece6b1a393b207e9f5ee274d133f92"}
Oct 30 16:42:47.536028 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:47.535986    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7440-458937" event={"ID":"e5aeb44f-aedb-4ed9-9eed-ae9764381724","Type":"ContainerStarted","Data":"ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5"}
Oct 30 16:42:47.570867 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:47.570803    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7440-458937" podStartSLOduration=1.897181116 podStartE2EDuration="4.570783666s" totalImagesPullingTime="2.67360255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:47.569147291 +0000 UTC m=+1197.423025556" watchObservedRunningTime="2026-10-30 16:42:47.570783666 +0000 UTC m=+1197.424661932"
Oct 30 16:42:48.459893 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:48.459859    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7440-458937"]
Oct 30 16:42:48.463395 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:48.463357    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7440-458937"]
Oct 30 16:42:49.548380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:49.548339    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7440-458937" podUID="e5aeb44f-aedb-4ed9-9eed-ae9764381724" containerName="test-curl-7440-458937" containerID="cri-o://ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5" gracePeriod=2
Oct 30 16:42:49.559421 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:49.559393    4497 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7440-458937\" is forbidden: User \"system:node:ip-10-0-136-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-74.ec2.internal' and this object" podUID="e5aeb44f-aedb-4ed9-9eed-ae9764381724" pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:50.307054 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:50.307018    4497 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7440-458937\" is forbidden: User \"system:node:ip-10-0-136-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-74.ec2.internal' and this object" podUID="e5aeb44f-aedb-4ed9-9eed-ae9764381724" pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:51.759471 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:51.759445    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:51.909709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:51.909620    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5aeb44f-aedb-4ed9-9eed-ae9764381724-kube-api-access-8fc9m\" (UniqueName: \"kubernetes.io/projected/e5aeb44f-aedb-4ed9-9eed-ae9764381724-kube-api-access-8fc9m\") pod \"e5aeb44f-aedb-4ed9-9eed-ae9764381724\" (UID: \"e5aeb44f-aedb-4ed9-9eed-ae9764381724\") "
Oct 30 16:42:51.911844 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:51.911815    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aeb44f-aedb-4ed9-9eed-ae9764381724-kube-api-access-8fc9m" pod "e5aeb44f-aedb-4ed9-9eed-ae9764381724" (UID: "e5aeb44f-aedb-4ed9-9eed-ae9764381724"). InnerVolumeSpecName "kube-api-access-8fc9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:42:51.924816 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:51.924778    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:42:52.012126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:52.012084    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fc9m\" (UniqueName: \"kubernetes.io/projected/e5aeb44f-aedb-4ed9-9eed-ae9764381724-kube-api-access-8fc9m\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:42:52.300595 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:52.300563    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5aeb44f-aedb-4ed9-9eed-ae9764381724" path="/var/lib/kubelet/pods/e5aeb44f-aedb-4ed9-9eed-ae9764381724/volumes"
Oct 30 16:42:52.569499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:52.569399    4497 generic.go:356] "Generic (PLEG): container finished" podID="e5aeb44f-aedb-4ed9-9eed-ae9764381724" containerID="ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5" exitCode=137
Oct 30 16:42:52.569499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:52.569444    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-458937"
Oct 30 16:42:52.569725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:52.569506    4497 scope.go:122] "RemoveContainer" containerID="ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5"
Oct 30 16:42:52.583176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:52.583155    4497 scope.go:122] "RemoveContainer" containerID="ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5"
Oct 30 16:42:52.583538 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:42:52.583514    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5\": container with ID starting with ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5 not found: ID does not exist" containerID="ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5"
Oct 30 16:42:52.583619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:52.583546    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5"} err="failed to get container status \"ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5\": rpc error: code = NotFound desc = could not find container \"ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5\": container with ID starting with ef837ea7a7bfa62bafcac9db9aaa185477d0bc56a8830676253f19c56a2e1ef5 not found: ID does not exist"
Oct 30 16:42:59.378209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:59.378030    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"]
Oct 30 16:42:59.528119 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:42:59.528079    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"]
Oct 30 16:43:04.379395 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.379302    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5" podUID="aa39be64-f7d1-4a1c-9e27-548948f1a0b6" containerName="payload-processing" containerID="cri-o://ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1" gracePeriod=25
Oct 30 16:43:04.530720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.530673    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg" podUID="768da413-799e-48c3-ae2c-c3bd651d018a" containerName="payload-pre-processing" containerID="cri-o://f15f21a5f5cc64947aad4a6b80a8f04196d469e900de630988152f8fd54c30e9" gracePeriod=25
Oct 30 16:43:04.689533 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.689509    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:43:04.727464 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.727417    4497 generic.go:356] "Generic (PLEG): container finished" podID="768da413-799e-48c3-ae2c-c3bd651d018a" containerID="f15f21a5f5cc64947aad4a6b80a8f04196d469e900de630988152f8fd54c30e9" exitCode=0
Oct 30 16:43:04.727614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.727592    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg" event={"ID":"768da413-799e-48c3-ae2c-c3bd651d018a","Type":"ContainerDied","Data":"f15f21a5f5cc64947aad4a6b80a8f04196d469e900de630988152f8fd54c30e9"}
Oct 30 16:43:04.733582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.729455    4497 generic.go:356] "Generic (PLEG): container finished" podID="aa39be64-f7d1-4a1c-9e27-548948f1a0b6" containerID="ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1" exitCode=0
Oct 30 16:43:04.733582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.729583    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5" event={"ID":"aa39be64-f7d1-4a1c-9e27-548948f1a0b6","Type":"ContainerDied","Data":"ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1"}
Oct 30 16:43:04.733582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.729622    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5" event={"ID":"aa39be64-f7d1-4a1c-9e27-548948f1a0b6","Type":"ContainerDied","Data":"d488ba3615b8f1d09ac2baaf4a66780c0b2ad987df2cdd9872e59e2ffe0cd184"}
Oct 30 16:43:04.733582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.729652    4497 scope.go:122] "RemoveContainer" containerID="ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1"
Oct 30 16:43:04.733582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.729811    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"
Oct 30 16:43:04.745244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.736406    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-kube-api-access-x6qrn\" (UniqueName: \"kubernetes.io/projected/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-kube-api-access-x6qrn\") pod \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\" (UID: \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\") "
Oct 30 16:43:04.745244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.736583    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-plugins-config-volume\") pod \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\" (UID: \"aa39be64-f7d1-4a1c-9e27-548948f1a0b6\") "
Oct 30 16:43:04.745244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.737385    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-plugins-config-volume" pod "aa39be64-f7d1-4a1c-9e27-548948f1a0b6" (UID: "aa39be64-f7d1-4a1c-9e27-548948f1a0b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:43:04.745244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.741352    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-kube-api-access-x6qrn" pod "aa39be64-f7d1-4a1c-9e27-548948f1a0b6" (UID: "aa39be64-f7d1-4a1c-9e27-548948f1a0b6"). InnerVolumeSpecName "kube-api-access-x6qrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:43:04.760491 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.760464    4497 scope.go:122] "RemoveContainer" containerID="ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1"
Oct 30 16:43:04.760935 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:04.760908    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1\": container with ID starting with ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1 not found: ID does not exist" containerID="ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1"
Oct 30 16:43:04.761022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.760955    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1"} err="failed to get container status \"ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1\": rpc error: code = NotFound desc = could not find container \"ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1\": container with ID starting with ec387e1f84acce5e824389a39fabaedef85100a1fcef183d6a27f21e9d7ec5a1 not found: ID does not exist"
Oct 30 16:43:04.815708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.815678    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:43:04.837602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.837573    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768da413-799e-48c3-ae2c-c3bd651d018a-kube-api-access-khbqd\" (UniqueName: \"kubernetes.io/projected/768da413-799e-48c3-ae2c-c3bd651d018a-kube-api-access-khbqd\") pod \"768da413-799e-48c3-ae2c-c3bd651d018a\" (UID: \"768da413-799e-48c3-ae2c-c3bd651d018a\") "
Oct 30 16:43:04.837757 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.837745    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768da413-799e-48c3-ae2c-c3bd651d018a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768da413-799e-48c3-ae2c-c3bd651d018a-plugins-config-volume\") pod \"768da413-799e-48c3-ae2c-c3bd651d018a\" (UID: \"768da413-799e-48c3-ae2c-c3bd651d018a\") "
Oct 30 16:43:04.838107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.838087    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:04.838107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.838085    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768da413-799e-48c3-ae2c-c3bd651d018a-plugins-config-volume" pod "768da413-799e-48c3-ae2c-c3bd651d018a" (UID: "768da413-799e-48c3-ae2c-c3bd651d018a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:43:04.838264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.838128    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6qrn\" (UniqueName: \"kubernetes.io/projected/aa39be64-f7d1-4a1c-9e27-548948f1a0b6-kube-api-access-x6qrn\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:04.839659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.839640    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768da413-799e-48c3-ae2c-c3bd651d018a-kube-api-access-khbqd" pod "768da413-799e-48c3-ae2c-c3bd651d018a" (UID: "768da413-799e-48c3-ae2c-c3bd651d018a"). InnerVolumeSpecName "kube-api-access-khbqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:43:04.939092 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.939009    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768da413-799e-48c3-ae2c-c3bd651d018a-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:04.939092 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:04.939046    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khbqd\" (UniqueName: \"kubernetes.io/projected/768da413-799e-48c3-ae2c-c3bd651d018a-kube-api-access-khbqd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:05.071911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.071877    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"]
Oct 30 16:43:05.077804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.077773    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-hxbh5"]
Oct 30 16:43:05.089350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.089316    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"]
Oct 30 16:43:05.089628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.089566    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx" podUID="4de7f113-46fc-4efd-b9ad-729351d065d9" containerName="maas-api" containerID="cri-o://9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca" gracePeriod=30
Oct 30 16:43:05.354887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.354857    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:43:05.547860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.547824    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de7f113-46fc-4efd-b9ad-729351d065d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7f113-46fc-4efd-b9ad-729351d065d9-openshift-service-ca\") pod \"4de7f113-46fc-4efd-b9ad-729351d065d9\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") "
Oct 30 16:43:05.548380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.547882    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-maas-api-tls\") pod \"4de7f113-46fc-4efd-b9ad-729351d065d9\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") "
Oct 30 16:43:05.548380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.547934    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de7f113-46fc-4efd-b9ad-729351d065d9-kube-api-access-pjfwl\" (UniqueName: \"kubernetes.io/projected/4de7f113-46fc-4efd-b9ad-729351d065d9-kube-api-access-pjfwl\") pod \"4de7f113-46fc-4efd-b9ad-729351d065d9\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") "
Oct 30 16:43:05.548380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.547982    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-metrics-certs\") pod \"4de7f113-46fc-4efd-b9ad-729351d065d9\" (UID: \"4de7f113-46fc-4efd-b9ad-729351d065d9\") "
Oct 30 16:43:05.548380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.548252    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de7f113-46fc-4efd-b9ad-729351d065d9-openshift-service-ca" pod "4de7f113-46fc-4efd-b9ad-729351d065d9" (UID: "4de7f113-46fc-4efd-b9ad-729351d065d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:43:05.548613 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.548547    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4de7f113-46fc-4efd-b9ad-729351d065d9-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:05.550248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.550218    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de7f113-46fc-4efd-b9ad-729351d065d9-kube-api-access-pjfwl" pod "4de7f113-46fc-4efd-b9ad-729351d065d9" (UID: "4de7f113-46fc-4efd-b9ad-729351d065d9"). InnerVolumeSpecName "kube-api-access-pjfwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:43:05.550535 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.550514    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-metrics-certs" pod "4de7f113-46fc-4efd-b9ad-729351d065d9" (UID: "4de7f113-46fc-4efd-b9ad-729351d065d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:43:05.550613 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.550584    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-maas-api-tls" pod "4de7f113-46fc-4efd-b9ad-729351d065d9" (UID: "4de7f113-46fc-4efd-b9ad-729351d065d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:43:05.649515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.649483    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:05.649515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.649514    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjfwl\" (UniqueName: \"kubernetes.io/projected/4de7f113-46fc-4efd-b9ad-729351d065d9-kube-api-access-pjfwl\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:05.649515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.649524    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4de7f113-46fc-4efd-b9ad-729351d065d9-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:05.742606 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.742570    4497 generic.go:356] "Generic (PLEG): container finished" podID="4de7f113-46fc-4efd-b9ad-729351d065d9" containerID="9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca" exitCode=0
Oct 30 16:43:05.742817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.742623    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"
Oct 30 16:43:05.742817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.742661    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx" event={"ID":"4de7f113-46fc-4efd-b9ad-729351d065d9","Type":"ContainerDied","Data":"9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca"}
Oct 30 16:43:05.742817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.742729    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx" event={"ID":"4de7f113-46fc-4efd-b9ad-729351d065d9","Type":"ContainerDied","Data":"ed078164958c9024e07441eae1d8a478318615e52d100fb36958d6221b674013"}
Oct 30 16:43:05.742817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.742755    4497 scope.go:122] "RemoveContainer" containerID="9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca"
Oct 30 16:43:05.745081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.745055    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg" event={"ID":"768da413-799e-48c3-ae2c-c3bd651d018a","Type":"ContainerDied","Data":"1bfcf77911be58952ac1925b89170c7d6b9f363f264aae6d61d6237b52b4635b"}
Oct 30 16:43:05.745326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.745159    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"
Oct 30 16:43:05.757383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.757364    4497 scope.go:122] "RemoveContainer" containerID="9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca"
Oct 30 16:43:05.757737 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:05.757714    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca\": container with ID starting with 9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca not found: ID does not exist" containerID="9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca"
Oct 30 16:43:05.757840 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.757744    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca"} err="failed to get container status \"9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca\": rpc error: code = NotFound desc = could not find container \"9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca\": container with ID starting with 9a5e55e98b9f72ec5320d3c30530ebaa09cc51e2ec1e11ef01cb1833e448d7ca not found: ID does not exist"
Oct 30 16:43:05.757840 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.757762    4497 scope.go:122] "RemoveContainer" containerID="f15f21a5f5cc64947aad4a6b80a8f04196d469e900de630988152f8fd54c30e9"
Oct 30 16:43:05.788037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.788008    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"]
Oct 30 16:43:05.803254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.803159    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx"]
Oct 30 16:43:05.846329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.846294    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"]
Oct 30 16:43:05.853047 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:05.853018    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-t2ssg"]
Oct 30 16:43:06.093265 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:06.088120    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"]
Oct 30 16:43:06.300953 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:06.300918    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de7f113-46fc-4efd-b9ad-729351d065d9" path="/var/lib/kubelet/pods/4de7f113-46fc-4efd-b9ad-729351d065d9/volumes"
Oct 30 16:43:06.301395 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:06.301380    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768da413-799e-48c3-ae2c-c3bd651d018a" path="/var/lib/kubelet/pods/768da413-799e-48c3-ae2c-c3bd651d018a/volumes"
Oct 30 16:43:06.301767 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:06.301746    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa39be64-f7d1-4a1c-9e27-548948f1a0b6" path="/var/lib/kubelet/pods/aa39be64-f7d1-4a1c-9e27-548948f1a0b6/volumes"
Oct 30 16:43:08.313009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:08.301483    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82381960-696b6bf757-bntsx" podUID="4de7f113-46fc-4efd-b9ad-729351d065d9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.96:8443/health\": context deadline exceeded"
Oct 30 16:43:11.088753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.088707    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24" podUID="b41536cd-79f3-4672-b457-e909c708b5bc" containerName="payload-processing" containerID="cri-o://6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54" gracePeriod=25
Oct 30 16:43:11.374965 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.374938    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:43:11.443707 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.443676    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41536cd-79f3-4672-b457-e909c708b5bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41536cd-79f3-4672-b457-e909c708b5bc-plugins-config-volume\") pod \"b41536cd-79f3-4672-b457-e909c708b5bc\" (UID: \"b41536cd-79f3-4672-b457-e909c708b5bc\") "
Oct 30 16:43:11.443889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.443789    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41536cd-79f3-4672-b457-e909c708b5bc-kube-api-access-tpj25\" (UniqueName: \"kubernetes.io/projected/b41536cd-79f3-4672-b457-e909c708b5bc-kube-api-access-tpj25\") pod \"b41536cd-79f3-4672-b457-e909c708b5bc\" (UID: \"b41536cd-79f3-4672-b457-e909c708b5bc\") "
Oct 30 16:43:11.444150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.444124    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41536cd-79f3-4672-b457-e909c708b5bc-plugins-config-volume" pod "b41536cd-79f3-4672-b457-e909c708b5bc" (UID: "b41536cd-79f3-4672-b457-e909c708b5bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:43:11.444282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.444266    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b41536cd-79f3-4672-b457-e909c708b5bc-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:11.446041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.446015    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41536cd-79f3-4672-b457-e909c708b5bc-kube-api-access-tpj25" pod "b41536cd-79f3-4672-b457-e909c708b5bc" (UID: "b41536cd-79f3-4672-b457-e909c708b5bc"). InnerVolumeSpecName "kube-api-access-tpj25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:43:11.546163 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.546121    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpj25\" (UniqueName: \"kubernetes.io/projected/b41536cd-79f3-4672-b457-e909c708b5bc-kube-api-access-tpj25\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:11.810696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.810661    4497 generic.go:356] "Generic (PLEG): container finished" podID="b41536cd-79f3-4672-b457-e909c708b5bc" containerID="6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54" exitCode=0
Oct 30 16:43:11.810899 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.810715    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"
Oct 30 16:43:11.810899 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.810729    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24" event={"ID":"b41536cd-79f3-4672-b457-e909c708b5bc","Type":"ContainerDied","Data":"6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54"}
Oct 30 16:43:11.810899 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.810774    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24" event={"ID":"b41536cd-79f3-4672-b457-e909c708b5bc","Type":"ContainerDied","Data":"d4dcac71d1fc26c75c78813bee4d9f49ea0ea2a08cc2659e15b66ecc683bc32b"}
Oct 30 16:43:11.810899 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.810796    4497 scope.go:122] "RemoveContainer" containerID="6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54"
Oct 30 16:43:11.826921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.823941    4497 scope.go:122] "RemoveContainer" containerID="6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54"
Oct 30 16:43:11.826921 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:11.824286    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54\": container with ID starting with 6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54 not found: ID does not exist" containerID="6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54"
Oct 30 16:43:11.826921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.824323    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54"} err="failed to get container status \"6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54\": rpc error: code = NotFound desc = could not find container \"6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54\": container with ID starting with 6a2e5bcad55901f6d03705ffb055632791db000b862afa5b120b6574d2dbcf54 not found: ID does not exist"
Oct 30 16:43:11.857423 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.857392    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"]
Oct 30 16:43:11.861999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:11.861973    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1189d4-f847fcc8b-qfc24"]
Oct 30 16:43:12.299935 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:12.299902    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41536cd-79f3-4672-b457-e909c708b5bc" path="/var/lib/kubelet/pods/b41536cd-79f3-4672-b457-e909c708b5bc/volumes"
Oct 30 16:43:21.916361 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:21.916324    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:43:27.533066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.533028    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"]
Oct 30 16:43:27.534012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.533978    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy"
Oct 30 16:43:27.534012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534003    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534019    4497 cpu_manager.go:414] "containerMap: removing container" podUID="768da413-799e-48c3-ae2c-c3bd651d018a" containerName="payload-pre-processing"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534026    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768da413-799e-48c3-ae2c-c3bd651d018a" containerName="payload-pre-processing"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534090    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e5aeb44f-aedb-4ed9-9eed-ae9764381724" containerName="test-curl-7440-458937"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534097    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5aeb44f-aedb-4ed9-9eed-ae9764381724" containerName="test-curl-7440-458937"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534109    4497 cpu_manager.go:414] "containerMap: removing container" podUID="aa39be64-f7d1-4a1c-9e27-548948f1a0b6" containerName="payload-processing"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534116    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa39be64-f7d1-4a1c-9e27-548948f1a0b6" containerName="payload-processing"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534138    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b41536cd-79f3-4672-b457-e909c708b5bc" containerName="payload-processing"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534145    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41536cd-79f3-4672-b457-e909c708b5bc" containerName="payload-processing"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534156    4497 cpu_manager.go:414] "containerMap: removing container" podUID="4de7f113-46fc-4efd-b9ad-729351d065d9" containerName="maas-api"
Oct 30 16:43:27.534205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534162    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de7f113-46fc-4efd-b9ad-729351d065d9" containerName="maas-api"
Oct 30 16:43:27.534708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534317    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5aeb44f-aedb-4ed9-9eed-ae9764381724" containerName="test-curl-7440-458937"
Oct 30 16:43:27.534708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534333    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa39be64-f7d1-4a1c-9e27-548948f1a0b6" containerName="payload-processing"
Oct 30 16:43:27.534708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534345    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41536cd-79f3-4672-b457-e909c708b5bc" containerName="payload-processing"
Oct 30 16:43:27.534708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534354    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de7f113-46fc-4efd-b9ad-729351d065d9" containerName="maas-api"
Oct 30 16:43:27.534708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534362    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="27051549-670c-4ee5-8a2f-4fee4f272067" containerName="istio-proxy"
Oct 30 16:43:27.534708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534370    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="768da413-799e-48c3-ae2c-c3bd651d018a" containerName="payload-pre-processing"
Oct 30 16:43:27.534708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.534386    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0fc41e-ab04-42e2-8f1f-301485dada69" containerName="istio-proxy"
Oct 30 16:43:27.541660 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.541637    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.564237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.564175    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0a98d1-openshift-default-dockercfg-hw8q6\""
Oct 30 16:43:27.568506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.568473    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"]
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.584594    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istiod-ca-cert\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.584699    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-data\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.584810    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-certs\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.584862    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-socket\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.584893    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-podinfo\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.584940    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-envoy\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.585024    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-token\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.585057    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdsj\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-kube-api-access-qwdsj\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.585232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.585180    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-credential-socket\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690225 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690063    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-credential-socket\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690225 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690217    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istiod-ca-cert\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690261    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-data\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690334    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-certs\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690373    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-socket\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690401    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-podinfo\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690431    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-envoy\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690516    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-token\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.690774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.690550    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdsj\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-kube-api-access-qwdsj\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.694217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.691411    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-credential-socket\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.694217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.692010    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istiod-ca-cert\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.694217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.692278    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-data\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.694217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.692506    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-certs\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.694217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.692735    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-socket\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.719066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.719007    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-podinfo\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.720845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.720816    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdsj\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-kube-api-access-qwdsj\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.721249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.721227    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-envoy\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.748090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.748057    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-token\") pod \"e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") " pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:27.856861 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:27.856760    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:28.206282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:28.206163    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"]
Oct 30 16:43:28.243490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:28.238144    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:43:28.243490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:28.238249    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:43:28.243490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:28.238297    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:43:28.925784 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:28.925748    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" event={"ID":"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce","Type":"ContainerStarted","Data":"600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb"}
Oct 30 16:43:28.926240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:28.925807    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" event={"ID":"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce","Type":"ContainerStarted","Data":"92824e864dbfc8112c20cfa08ae4ce39ce95eca46807a491771dab587a81bd53"}
Oct 30 16:43:28.967068 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:28.967011    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" podStartSLOduration=1.96699533 podStartE2EDuration="1.96699533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:28.965354183 +0000 UTC m=+1238.819232447" watchObservedRunningTime="2026-10-30 16:43:28.96699533 +0000 UTC m=+1238.820873594"
Oct 30 16:43:29.860925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:29.857312    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:29.874446 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:29.874410    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:29.935230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:29.935158    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:29.936102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:29.936080    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:43:33.318699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.318665    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"]
Oct 30 16:43:33.319126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.318875    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy" containerID="cri-o://b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948" gracePeriod=30
Oct 30 16:43:33.420220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.420170    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"]
Oct 30 16:43:33.424539 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.424514    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:33.437343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.437321    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0a98d1\""
Oct 30 16:43:33.437469 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.437397    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0a98d1-dockercfg-zhfcf\""
Oct 30 16:43:33.445106 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.445082    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"]
Oct 30 16:43:33.499002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.498967    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"]
Oct 30 16:43:33.514925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.514534    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plcwl\" (UniqueName: \"kubernetes.io/projected/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-kube-api-access-plcwl\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk\" (UID: \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:33.514925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.514718    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk\" (UID: \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:33.517886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.517847    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:33.561207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.560260    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"]
Oct 30 16:43:33.616411 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.616381    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plcwl\" (UniqueName: \"kubernetes.io/projected/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-kube-api-access-plcwl\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk\" (UID: \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:33.616571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.616510    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk\" (UID: \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:33.616650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.616591    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3f14dc-d400-4cb8-8460-112db7bc0abe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g\" (UID: \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:33.616650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.616626    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xqx\" (UniqueName: \"kubernetes.io/projected/bc3f14dc-d400-4cb8-8460-112db7bc0abe-kube-api-access-j2xqx\") pod \"payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g\" (UID: \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:33.617612 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.617577    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk\" (UID: \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:33.639349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.639316    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plcwl\" (UniqueName: \"kubernetes.io/projected/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-kube-api-access-plcwl\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk\" (UID: \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:33.717734 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.717701    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3f14dc-d400-4cb8-8460-112db7bc0abe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g\" (UID: \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:33.717949 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.717777    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xqx\" (UniqueName: \"kubernetes.io/projected/bc3f14dc-d400-4cb8-8460-112db7bc0abe-kube-api-access-j2xqx\") pod \"payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g\" (UID: \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:33.718749 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.718721    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3f14dc-d400-4cb8-8460-112db7bc0abe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g\" (UID: \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:33.738957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.738891    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:33.739273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.739252    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xqx\" (UniqueName: \"kubernetes.io/projected/bc3f14dc-d400-4cb8-8460-112db7bc0abe-kube-api-access-j2xqx\") pod \"payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g\" (UID: \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:33.913574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:33.913491    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:34.074663 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:34.072878    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"]
Oct 30 16:43:34.075648 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:43:34.075528    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ee8c42_1e5a_4c98_a798_0f3d986e04e3.slice/crio-fae2e7cb439de23c06d0751c9955777608121b2a457fb56a8e7a9bb562c2b6c4 WatchSource:0}: Error finding container fae2e7cb439de23c06d0751c9955777608121b2a457fb56a8e7a9bb562c2b6c4: Status 404 returned error can't find the container with id fae2e7cb439de23c06d0751c9955777608121b2a457fb56a8e7a9bb562c2b6c4
Oct 30 16:43:34.222128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:34.222051    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"]
Oct 30 16:43:34.225150 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:43:34.225125    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3f14dc_d400_4cb8_8460_112db7bc0abe.slice/crio-8e8979a79084bef92f3e514373858385aff6abb0b97f51243013d8e9ee1d984b WatchSource:0}: Error finding container 8e8979a79084bef92f3e514373858385aff6abb0b97f51243013d8e9ee1d984b: Status 404 returned error can't find the container with id 8e8979a79084bef92f3e514373858385aff6abb0b97f51243013d8e9ee1d984b
Oct 30 16:43:34.982393 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:34.982356    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g" event={"ID":"bc3f14dc-d400-4cb8-8460-112db7bc0abe","Type":"ContainerStarted","Data":"631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b"}
Oct 30 16:43:34.983060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:34.983035    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g" event={"ID":"bc3f14dc-d400-4cb8-8460-112db7bc0abe","Type":"ContainerStarted","Data":"8e8979a79084bef92f3e514373858385aff6abb0b97f51243013d8e9ee1d984b"}
Oct 30 16:43:34.983468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:34.983449    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:34.984791 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:34.984754    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk" event={"ID":"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3","Type":"ContainerStarted","Data":"1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e"}
Oct 30 16:43:34.984903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:34.984800    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk" event={"ID":"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3","Type":"ContainerStarted","Data":"fae2e7cb439de23c06d0751c9955777608121b2a457fb56a8e7a9bb562c2b6c4"}
Oct 30 16:43:34.984903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:34.984844    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:35.024716 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.024662    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g" podStartSLOduration=2.024646333 podStartE2EDuration="2.024646333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:35.021522665 +0000 UTC m=+1244.875400931" watchObservedRunningTime="2026-10-30 16:43:35.024646333 +0000 UTC m=+1244.878524596"
Oct 30 16:43:35.060846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.060798    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk" podStartSLOduration=2.060784462 podStartE2EDuration="2.060784462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:35.060440207 +0000 UTC m=+1244.914318472" watchObservedRunningTime="2026-10-30 16:43:35.060784462 +0000 UTC m=+1244.914662726"
Oct 30 16:43:35.196720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.196687    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"]
Oct 30 16:43:35.200967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.200944    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:35.213222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.213197    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-843ab3\""
Oct 30 16:43:35.223501 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.223478    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"]
Oct 30 16:43:35.251897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.251822    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"]
Oct 30 16:43:35.256252 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.256223    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.268673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.268636    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-843ab3-dockercfg-w2rs8\""
Oct 30 16:43:35.276949 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.276915    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"]
Oct 30 16:43:35.351233 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.351170    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac197c92-563f-418c-9dd7-0275e17a254b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558\" (UID: \"ac197c92-563f-418c-9dd7-0275e17a254b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:35.351441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.351246    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289679a3-d78c-45c5-90dd-2a36bfdf19de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm\" (UID: \"289679a3-d78c-45c5-90dd-2a36bfdf19de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.351441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.351337    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hbb2\" (UniqueName: \"kubernetes.io/projected/289679a3-d78c-45c5-90dd-2a36bfdf19de-kube-api-access-6hbb2\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm\" (UID: \"289679a3-d78c-45c5-90dd-2a36bfdf19de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.351584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.351442    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvsbf\" (UniqueName: \"kubernetes.io/projected/ac197c92-563f-418c-9dd7-0275e17a254b-kube-api-access-gvsbf\") pod \"payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558\" (UID: \"ac197c92-563f-418c-9dd7-0275e17a254b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:35.460207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.458010    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac197c92-563f-418c-9dd7-0275e17a254b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558\" (UID: \"ac197c92-563f-418c-9dd7-0275e17a254b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:35.460207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.459161    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289679a3-d78c-45c5-90dd-2a36bfdf19de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm\" (UID: \"289679a3-d78c-45c5-90dd-2a36bfdf19de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.460207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.459315    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hbb2\" (UniqueName: \"kubernetes.io/projected/289679a3-d78c-45c5-90dd-2a36bfdf19de-kube-api-access-6hbb2\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm\" (UID: \"289679a3-d78c-45c5-90dd-2a36bfdf19de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.460207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.459077    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac197c92-563f-418c-9dd7-0275e17a254b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558\" (UID: \"ac197c92-563f-418c-9dd7-0275e17a254b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:35.460207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.460129    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289679a3-d78c-45c5-90dd-2a36bfdf19de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm\" (UID: \"289679a3-d78c-45c5-90dd-2a36bfdf19de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.464206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.460974    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvsbf\" (UniqueName: \"kubernetes.io/projected/ac197c92-563f-418c-9dd7-0275e17a254b-kube-api-access-gvsbf\") pod \"payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558\" (UID: \"ac197c92-563f-418c-9dd7-0275e17a254b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:35.481679 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.481643    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hbb2\" (UniqueName: \"kubernetes.io/projected/289679a3-d78c-45c5-90dd-2a36bfdf19de-kube-api-access-6hbb2\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm\" (UID: \"289679a3-d78c-45c5-90dd-2a36bfdf19de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.482349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.482329    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvsbf\" (UniqueName: \"kubernetes.io/projected/ac197c92-563f-418c-9dd7-0275e17a254b-kube-api-access-gvsbf\") pod \"payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558\" (UID: \"ac197c92-563f-418c-9dd7-0275e17a254b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:35.516431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.516397    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:35.574714 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.574685    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.826296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.826272    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"]
Oct 30 16:43:35.828855 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:43:35.828831    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac197c92_563f_418c_9dd7_0275e17a254b.slice/crio-44d92d06e839ed88da3003a4952c19bcc23e7ee2da7edda04ea02fbba6ea49b0 WatchSource:0}: Error finding container 44d92d06e839ed88da3003a4952c19bcc23e7ee2da7edda04ea02fbba6ea49b0: Status 404 returned error can't find the container with id 44d92d06e839ed88da3003a4952c19bcc23e7ee2da7edda04ea02fbba6ea49b0
Oct 30 16:43:35.912622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.912602    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"]
Oct 30 16:43:35.915535 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:43:35.915507    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289679a3_d78c_45c5_90dd_2a36bfdf19de.slice/crio-f32f0d2499b3e01f528dc6b5f613b788ae5a5ede54262766eb274ee5f942bc5d WatchSource:0}: Error finding container f32f0d2499b3e01f528dc6b5f613b788ae5a5ede54262766eb274ee5f942bc5d: Status 404 returned error can't find the container with id f32f0d2499b3e01f528dc6b5f613b788ae5a5ede54262766eb274ee5f942bc5d
Oct 30 16:43:35.994487 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.994450    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm" event={"ID":"289679a3-d78c-45c5-90dd-2a36bfdf19de","Type":"ContainerStarted","Data":"a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f"}
Oct 30 16:43:35.994941 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.994522    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:35.994941 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.994549    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm" event={"ID":"289679a3-d78c-45c5-90dd-2a36bfdf19de","Type":"ContainerStarted","Data":"f32f0d2499b3e01f528dc6b5f613b788ae5a5ede54262766eb274ee5f942bc5d"}
Oct 30 16:43:35.996210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.996158    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558" event={"ID":"ac197c92-563f-418c-9dd7-0275e17a254b","Type":"ContainerStarted","Data":"f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668"}
Oct 30 16:43:35.996314 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:35.996236    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558" event={"ID":"ac197c92-563f-418c-9dd7-0275e17a254b","Type":"ContainerStarted","Data":"44d92d06e839ed88da3003a4952c19bcc23e7ee2da7edda04ea02fbba6ea49b0"}
Oct 30 16:43:36.035220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:36.035149    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm" podStartSLOduration=1.035132747 podStartE2EDuration="1.035132747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:36.031256117 +0000 UTC m=+1245.885134380" watchObservedRunningTime="2026-10-30 16:43:36.035132747 +0000 UTC m=+1245.889011010"
Oct 30 16:43:36.076036 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:36.075935    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558" podStartSLOduration=1.075922813 podStartE2EDuration="1.075922813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:36.071573795 +0000 UTC m=+1245.925452059" watchObservedRunningTime="2026-10-30 16:43:36.075922813 +0000 UTC m=+1245.929801077"
Oct 30 16:43:36.949477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:36.949436    4497 patch_prober.go:29] interesting pod/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:43:36.949639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:36.949505    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:43:37.003326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:37.003291    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:38.595144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.595080    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:43:38.709923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.709889    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-credential-socket\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.709951    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6738852d-9876-451d-8b86-9718d8eb47c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6738852d-9876-451d-8b86-9718d8eb47c4-istiod-ca-cert\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.709978    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-certs\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710055    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-istio-token\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-istio-token\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710090    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-data\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710132    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6738852d-9876-451d-8b86-9718d8eb47c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6738852d-9876-451d-8b86-9718d8eb47c4-istio-podinfo\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710175    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-socket\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710306    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-kube-api-access-k2w6j\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-kube-api-access-k2w6j\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710523 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710320    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-credential-socket" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:43:38.710523 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710366    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6738852d-9876-451d-8b86-9718d8eb47c4-istiod-ca-cert" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:43:38.710523 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710370    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-envoy\") pod \"6738852d-9876-451d-8b86-9718d8eb47c4\" (UID: \"6738852d-9876-451d-8b86-9718d8eb47c4\") "
Oct 30 16:43:38.710677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710400    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-certs" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:43:38.710677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710444    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-data" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:43:38.710677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.710538    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-socket" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:43:38.711213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.711175    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:38.711268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.711218    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6738852d-9876-451d-8b86-9718d8eb47c4-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:38.711268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.711233    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:38.711268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.711246    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:38.711268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.711260    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:38.712318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.712296    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-istio-token" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:43:38.712550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.712521    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-kube-api-access-k2w6j" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "kube-api-access-k2w6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:43:38.712846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.712825    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6738852d-9876-451d-8b86-9718d8eb47c4-istio-podinfo" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:43:38.712917 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.712829    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-envoy" pod "6738852d-9876-451d-8b86-9718d8eb47c4" (UID: "6738852d-9876-451d-8b86-9718d8eb47c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:43:38.812604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.812562    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:38.812604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.812594    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6738852d-9876-451d-8b86-9718d8eb47c4-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:38.812604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.812608    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2w6j\" (UniqueName: \"kubernetes.io/projected/6738852d-9876-451d-8b86-9718d8eb47c4-kube-api-access-k2w6j\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:38.812830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:38.812620    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6738852d-9876-451d-8b86-9718d8eb47c4-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:43:39.017199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.017155    4497 generic.go:356] "Generic (PLEG): container finished" podID="6738852d-9876-451d-8b86-9718d8eb47c4" containerID="b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948" exitCode=0
Oct 30 16:43:39.017362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.017217    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"
Oct 30 16:43:39.017362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.017238    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" event={"ID":"6738852d-9876-451d-8b86-9718d8eb47c4","Type":"ContainerDied","Data":"b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948"}
Oct 30 16:43:39.017362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.017318    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws" event={"ID":"6738852d-9876-451d-8b86-9718d8eb47c4","Type":"ContainerDied","Data":"107fd2fa642979f1649f2f9d6bde1465515c183d2c0f0d1d7ff980081ffc280b"}
Oct 30 16:43:39.017362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.017345    4497 scope.go:122] "RemoveContainer" containerID="b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948"
Oct 30 16:43:39.027990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.027963    4497 scope.go:122] "RemoveContainer" containerID="de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e"
Oct 30 16:43:39.037588 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.037571    4497 scope.go:122] "RemoveContainer" containerID="b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948"
Oct 30 16:43:39.037888 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:39.037869    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948\": container with ID starting with b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948 not found: ID does not exist" containerID="b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948"
Oct 30 16:43:39.037934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.037898    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948"} err="failed to get container status \"b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948\": rpc error: code = NotFound desc = could not find container \"b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948\": container with ID starting with b56cb55dd2555bf8448371d92244ad2fa501d8e208b9e5059129c9c9de992948 not found: ID does not exist"
Oct 30 16:43:39.037934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.037915    4497 scope.go:122] "RemoveContainer" containerID="de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e"
Oct 30 16:43:39.038134 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:39.038115    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e\": container with ID starting with de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e not found: ID does not exist" containerID="de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e"
Oct 30 16:43:39.038171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.038139    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e"} err="failed to get container status \"de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e\": rpc error: code = NotFound desc = could not find container \"de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e\": container with ID starting with de616a506220221f6107e8dcaa73b7f4c26de884d601b9448826190dbb507b1e not found: ID does not exist"
Oct 30 16:43:39.063611 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.063579    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"]
Oct 30 16:43:39.072732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:39.072707    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1189d4-openshift-default-5d45dffcd5-r9bws"]
Oct 30 16:43:40.302582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:40.302554    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" path="/var/lib/kubelet/pods/6738852d-9876-451d-8b86-9718d8eb47c4/volumes"
Oct 30 16:43:41.168906 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.168864    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"]
Oct 30 16:43:41.170414 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.170387    4497 cpu_manager.go:414] "containerMap: removing container" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy"
Oct 30 16:43:41.170414 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.170412    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy"
Oct 30 16:43:41.170589 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.170442    4497 cpu_manager.go:414] "containerMap: removing container" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy"
Oct 30 16:43:41.170589 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.170450    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy"
Oct 30 16:43:41.170734 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.170719    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy"
Oct 30 16:43:41.170797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.170747    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="6738852d-9876-451d-8b86-9718d8eb47c4" containerName="istio-proxy"
Oct 30 16:43:41.176631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.176606    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:41.187613 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.187578    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"]
Oct 30 16:43:41.192638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.192615    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7c23a4\""
Oct 30 16:43:41.246645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.246608    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"]
Oct 30 16:43:41.250985 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.250968    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:41.267229 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.267174    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7c23a4-dockercfg-rnt4z\""
Oct 30 16:43:41.271251 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.271226    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"]
Oct 30 16:43:41.394219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.393494    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng\" (UID: \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:41.394219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.393597    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqdf\" (UniqueName: \"kubernetes.io/projected/3a1a1287-4249-465f-be17-795d3ee29d0b-kube-api-access-kpqdf\") pod \"payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw\" (UID: \"3a1a1287-4249-465f-be17-795d3ee29d0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:41.394219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.393626    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx644\" (UniqueName: \"kubernetes.io/projected/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-kube-api-access-lx644\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng\" (UID: \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:41.394219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.393818    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1a1287-4249-465f-be17-795d3ee29d0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw\" (UID: \"3a1a1287-4249-465f-be17-795d3ee29d0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:41.499126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.499020    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1a1287-4249-465f-be17-795d3ee29d0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw\" (UID: \"3a1a1287-4249-465f-be17-795d3ee29d0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:41.499665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.499632    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng\" (UID: \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:41.499786 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.499754    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqdf\" (UniqueName: \"kubernetes.io/projected/3a1a1287-4249-465f-be17-795d3ee29d0b-kube-api-access-kpqdf\") pod \"payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw\" (UID: \"3a1a1287-4249-465f-be17-795d3ee29d0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:41.499867 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.499786    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx644\" (UniqueName: \"kubernetes.io/projected/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-kube-api-access-lx644\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng\" (UID: \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:41.500925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.500872    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1a1287-4249-465f-be17-795d3ee29d0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw\" (UID: \"3a1a1287-4249-465f-be17-795d3ee29d0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:41.501220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.501172    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng\" (UID: \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:41.524066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.524025    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqdf\" (UniqueName: \"kubernetes.io/projected/3a1a1287-4249-465f-be17-795d3ee29d0b-kube-api-access-kpqdf\") pod \"payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw\" (UID: \"3a1a1287-4249-465f-be17-795d3ee29d0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:41.525118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.525096    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx644\" (UniqueName: \"kubernetes.io/projected/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-kube-api-access-lx644\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng\" (UID: \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:41.565229 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.565175    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:41.793466 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.793430    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:41.885517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:41.885482    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"]
Oct 30 16:43:41.889738 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:43:41.889692    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5947e42_d9ea_43c6_b3cf_0b5fd24f5e85.slice/crio-a5c3fe28f59a5bd1a026507b657d65dd57c47489fd97cb41c2e3b7876312e2b7 WatchSource:0}: Error finding container a5c3fe28f59a5bd1a026507b657d65dd57c47489fd97cb41c2e3b7876312e2b7: Status 404 returned error can't find the container with id a5c3fe28f59a5bd1a026507b657d65dd57c47489fd97cb41c2e3b7876312e2b7
Oct 30 16:43:42.046437 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:42.046338    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng" event={"ID":"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85","Type":"ContainerStarted","Data":"efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46"}
Oct 30 16:43:42.046437 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:42.046420    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:42.046637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:42.046450    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng" event={"ID":"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85","Type":"ContainerStarted","Data":"a5c3fe28f59a5bd1a026507b657d65dd57c47489fd97cb41c2e3b7876312e2b7"}
Oct 30 16:43:42.087545 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:42.087490    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng" podStartSLOduration=1.087474117 podStartE2EDuration="1.087474117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:42.086954347 +0000 UTC m=+1251.940832611" watchObservedRunningTime="2026-10-30 16:43:42.087474117 +0000 UTC m=+1251.941352381"
Oct 30 16:43:42.110457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:42.108433    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"]
Oct 30 16:43:42.112996 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:43:42.112961    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1a1287_4249_465f_be17_795d3ee29d0b.slice/crio-9d1718be62424cce3cf50ba54fa60affa04036ce004ae09139908b207baf4d4b WatchSource:0}: Error finding container 9d1718be62424cce3cf50ba54fa60affa04036ce004ae09139908b207baf4d4b: Status 404 returned error can't find the container with id 9d1718be62424cce3cf50ba54fa60affa04036ce004ae09139908b207baf4d4b
Oct 30 16:43:43.054478 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.054442    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw" event={"ID":"3a1a1287-4249-465f-be17-795d3ee29d0b","Type":"ContainerStarted","Data":"3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a"}
Oct 30 16:43:43.054929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.054498    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw" event={"ID":"3a1a1287-4249-465f-be17-795d3ee29d0b","Type":"ContainerStarted","Data":"9d1718be62424cce3cf50ba54fa60affa04036ce004ae09139908b207baf4d4b"}
Oct 30 16:43:43.054929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.054722    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:43.100056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.100001    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw" podStartSLOduration=2.099986807 podStartE2EDuration="2.099986807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:43.096927077 +0000 UTC m=+1252.950805342" watchObservedRunningTime="2026-10-30 16:43:43.099986807 +0000 UTC m=+1252.953865072"
Oct 30 16:43:43.800551 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.800516    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"]
Oct 30 16:43:43.804807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.804783    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.820279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.820257    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0a98d1-serving-cert\""
Oct 30 16:43:43.835639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.834227    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"]
Oct 30 16:43:43.835639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.835217    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.835639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.835284    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.835639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.835396    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcczm\" (UniqueName: \"kubernetes.io/projected/4cbc1604-9a77-4f6e-adea-2605b83d21c5-kube-api-access-qcczm\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.835639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.835430    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbc1604-9a77-4f6e-adea-2605b83d21c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.936685 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.936656    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.936873 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.936713    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.936873 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.936820    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcczm\" (UniqueName: \"kubernetes.io/projected/4cbc1604-9a77-4f6e-adea-2605b83d21c5-kube-api-access-qcczm\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.936873 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.936855    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbc1604-9a77-4f6e-adea-2605b83d21c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.937039 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:43.936879    4497 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-serving-cert: secret "maas-api-e2e-ipp-a-0a98d1-serving-cert" not found
Oct 30 16:43:43.937039 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:43.936962    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls podName:4cbc1604-9a77-4f6e-adea-2605b83d21c5 nodeName:}" failed. No retries permitted until 2026-10-30 16:43:44.436939089 +0000 UTC m=+1254.290817346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls") pod "maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr" (UID: "4cbc1604-9a77-4f6e-adea-2605b83d21c5") : secret "maas-api-e2e-ipp-a-0a98d1-serving-cert" not found
Oct 30 16:43:43.937772 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.937752    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbc1604-9a77-4f6e-adea-2605b83d21c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.939465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.939442    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-metrics-certs\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:43.957982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:43.957956    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcczm\" (UniqueName: \"kubernetes.io/projected/4cbc1604-9a77-4f6e-adea-2605b83d21c5-kube-api-access-qcczm\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:44.442995 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:44.442959    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:44.445396 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:44.445375    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:44.719698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:44.719598    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:45.078209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:45.075506    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"]
Oct 30 16:43:45.100439 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:43:45.100406    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbc1604_9a77_4f6e_adea_2605b83d21c5.slice/crio-54631a9537aee642bbbbf170ac33896d8b4b60ea093795030e91fbae48514f6d WatchSource:0}: Error finding container 54631a9537aee642bbbbf170ac33896d8b4b60ea093795030e91fbae48514f6d: Status 404 returned error can't find the container with id 54631a9537aee642bbbbf170ac33896d8b4b60ea093795030e91fbae48514f6d
Oct 30 16:43:45.999886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:45.999810    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:43:46.113590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:46.113559    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr" event={"ID":"4cbc1604-9a77-4f6e-adea-2605b83d21c5","Type":"ContainerStarted","Data":"ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b"}
Oct 30 16:43:46.113767 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:46.113613    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr" event={"ID":"4cbc1604-9a77-4f6e-adea-2605b83d21c5","Type":"ContainerStarted","Data":"54631a9537aee642bbbbf170ac33896d8b4b60ea093795030e91fbae48514f6d"}
Oct 30 16:43:46.113767 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:46.113712    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:46.152559 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:46.152506    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr" podStartSLOduration=2.690376938 podStartE2EDuration="3.152491246s" totalImagesPullingTime="462.114308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:46.151385588 +0000 UTC m=+1256.005263854" watchObservedRunningTime="2026-10-30 16:43:46.152491246 +0000 UTC m=+1256.006369510"
Oct 30 16:43:47.005988 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:47.005956    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:43:47.006415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:47.006043    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:43:48.014453 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:48.014419    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:43:51.936430 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:43:51.936368    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:43:52.133905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:52.133869    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:43:53.057814 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:53.057778    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:43:54.064802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:54.064776    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:43:59.337790 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.337752    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"]
Oct 30 16:43:59.344701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.344678    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.366976 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.366944    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-08fcc8-openshift-default-dockercfg-7fkrk\""
Oct 30 16:43:59.404792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.404762    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"]
Oct 30 16:43:59.456428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.456385    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-certs\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.456835 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.456769    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-token\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.457019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.456995    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-podinfo\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.457238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.457221    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-envoy\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.457450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.457432    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-credential-socket\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.457718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.457683    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-data\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.458052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.457986    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-socket\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.458319 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.458299    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqsz\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-kube-api-access-5mqsz\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.458604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.458586    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istiod-ca-cert\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560096    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqsz\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-kube-api-access-5mqsz\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560165    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istiod-ca-cert\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560318    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-certs\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560359    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-token\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560400    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-podinfo\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560432    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-envoy\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560477    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-credential-socket\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560573    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-data\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.560701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.560645    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-socket\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.561149 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.561119    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-socket\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.562541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.562517    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istiod-ca-cert\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.563379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.563264    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-credential-socket\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.563541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.563330    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-data\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.591860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.583075    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-podinfo\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.591860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.588229    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-certs\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.603260 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.595675    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-envoy\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.605747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.605717    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-token\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.606779 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.606696    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqsz\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-kube-api-access-5mqsz\") pod \"e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") " pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:43:59.664208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:43:59.664142    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:44:00.029018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.028991    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"]
Oct 30 16:44:00.031969 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:44:00.031941    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f1d7ae_9027_4ee7_8972_bc953e335c04.slice/crio-5376636552b21bb259d2c5d6e94731ff6de372c7767ccf52f69ea742c864b32e WatchSource:0}: Error finding container 5376636552b21bb259d2c5d6e94731ff6de372c7767ccf52f69ea742c864b32e: Status 404 returned error can't find the container with id 5376636552b21bb259d2c5d6e94731ff6de372c7767ccf52f69ea742c864b32e
Oct 30 16:44:00.034718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.034680    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:00.034826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.034753    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:00.034826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.034787    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:00.254330 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.254285    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" event={"ID":"f4f1d7ae-9027-4ee7-8972-bc953e335c04","Type":"ContainerStarted","Data":"b0e69a0db9afaaaff31d18bb4c1d07140cf7cc1cfdda866063e091568ee0ce1e"}
Oct 30 16:44:00.254520 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.254346    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" event={"ID":"f4f1d7ae-9027-4ee7-8972-bc953e335c04","Type":"ContainerStarted","Data":"5376636552b21bb259d2c5d6e94731ff6de372c7767ccf52f69ea742c864b32e"}
Oct 30 16:44:00.521235 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.518060    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podStartSLOduration=1.518034767 podStartE2EDuration="1.518034767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:00.32452267 +0000 UTC m=+1270.178400946" watchObservedRunningTime="2026-10-30 16:44:00.518034767 +0000 UTC m=+1270.371913034"
Oct 30 16:44:00.530459 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.522249    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"]
Oct 30 16:44:00.530459 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.522877    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy" containerID="cri-o://94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323" gracePeriod=30
Oct 30 16:44:00.664625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:00.664587    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:44:01.670891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:01.670859    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:44:02.269171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:02.269138    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:44:02.270281 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:02.270260    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:44:05.806263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.806238    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:44:05.810359 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810339    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-token\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-token\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.810427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810374    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3099ed01-f441-4de2-bb5b-cbc6d167c809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3099ed01-f441-4de2-bb5b-cbc6d167c809-istiod-ca-cert\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.810427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810422    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-podinfo\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.810498 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810450    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-socket\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.810562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810542    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-envoy\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.810632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810591    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-credential-socket\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.810690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810649    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-data\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.810690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810681    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-kube-api-access-rcflp\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-kube-api-access-rcflp\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.810806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810773    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-socket" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:05.810806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810793    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3099ed01-f441-4de2-bb5b-cbc6d167c809-istiod-ca-cert" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:05.810919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810876    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-certs\") pod \"3099ed01-f441-4de2-bb5b-cbc6d167c809\" (UID: \"3099ed01-f441-4de2-bb5b-cbc6d167c809\") "
Oct 30 16:44:05.811013 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.810990    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-credential-socket" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:05.811124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.811105    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-data" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:05.811329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.811284    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-certs" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:05.811892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.811866    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:05.811978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.811897    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:05.811978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.811913    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:05.811978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.811928    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3099ed01-f441-4de2-bb5b-cbc6d167c809-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:05.811978 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.811941    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:05.812830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.812803    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-token" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:05.812938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.812920    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-envoy" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:05.812998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.812968    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-podinfo" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:44:05.813048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.813004    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-kube-api-access-rcflp" pod "3099ed01-f441-4de2-bb5b-cbc6d167c809" (UID: "3099ed01-f441-4de2-bb5b-cbc6d167c809"). InnerVolumeSpecName "kube-api-access-rcflp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:05.913339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.913294    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:05.913339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.913329    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:05.913339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.913339    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcflp\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-kube-api-access-rcflp\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:05.913339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:05.913350    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3099ed01-f441-4de2-bb5b-cbc6d167c809-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:06.299815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.299781    4497 generic.go:356] "Generic (PLEG): container finished" podID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerID="600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb" exitCode=0
Oct 30 16:44:06.301398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.301364    4497 generic.go:356] "Generic (PLEG): container finished" podID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerID="94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323" exitCode=0
Oct 30 16:44:06.301537 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.301500    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"
Oct 30 16:44:06.301702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.301679    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" event={"ID":"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce","Type":"ContainerDied","Data":"600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb"}
Oct 30 16:44:06.301793 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.301772    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" event={"ID":"3099ed01-f441-4de2-bb5b-cbc6d167c809","Type":"ContainerDied","Data":"94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323"}
Oct 30 16:44:06.301992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.301969    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv" event={"ID":"3099ed01-f441-4de2-bb5b-cbc6d167c809","Type":"ContainerDied","Data":"2c43fc60a784594e986481fa8a6363863eb2011ca75c2e543cd3b730ea693b78"}
Oct 30 16:44:06.302097 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.302040    4497 scope.go:122] "RemoveContainer" containerID="94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323"
Oct 30 16:44:06.302537 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.302516    4497 scope.go:122] "RemoveContainer" containerID="600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb"
Oct 30 16:44:06.310231 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.310172    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:06.310328 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.310290    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:06.310368 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.310342    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:06.322941 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.322901    4497 scope.go:122] "RemoveContainer" containerID="94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323"
Oct 30 16:44:06.323903 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:06.323873    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323\": container with ID starting with 94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323 not found: ID does not exist" containerID="94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323"
Oct 30 16:44:06.324091 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.324051    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323"} err="failed to get container status \"94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323\": rpc error: code = NotFound desc = could not find container \"94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323\": container with ID starting with 94755646cca6bc03a656ccee71a9abe700497aae17dcd60f1756c564d70db323 not found: ID does not exist"
Oct 30 16:44:06.355803 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.355773    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"]
Oct 30 16:44:06.363310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.363280    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w7ctv"]
Oct 30 16:44:06.409886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.409858    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"]
Oct 30 16:44:06.410547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.410532    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy"
Oct 30 16:44:06.410599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.410548    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy"
Oct 30 16:44:06.410665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.410655    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" containerName="istio-proxy"
Oct 30 16:44:06.413032 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.413013    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:06.429477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.429448    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-08fcc8\""
Oct 30 16:44:06.446171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.446138    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"]
Oct 30 16:44:06.469472 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.469434    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"]
Oct 30 16:44:06.474114 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.474090    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:06.492263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.492235    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-08fcc8-dockercfg-4kcgd\""
Oct 30 16:44:06.512098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.512063    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"]
Oct 30 16:44:06.521927 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.521888    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kpq\" (UniqueName: \"kubernetes.io/projected/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-kube-api-access-d5kpq\") pod \"payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67\" (UID: \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:06.522110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.522087    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67\" (UID: \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:06.522270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.522247    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjlw\" (UniqueName: \"kubernetes.io/projected/f8aba258-c5a6-4e67-9c6f-0037d6a61193-kube-api-access-xjjlw\") pod \"payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j\" (UID: \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\") " pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:06.522359 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.522332    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8aba258-c5a6-4e67-9c6f-0037d6a61193-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j\" (UID: \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\") " pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:06.624266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.624117    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjlw\" (UniqueName: \"kubernetes.io/projected/f8aba258-c5a6-4e67-9c6f-0037d6a61193-kube-api-access-xjjlw\") pod \"payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j\" (UID: \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\") " pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:06.624266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.624212    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8aba258-c5a6-4e67-9c6f-0037d6a61193-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j\" (UID: \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\") " pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:06.624504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.624409    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kpq\" (UniqueName: \"kubernetes.io/projected/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-kube-api-access-d5kpq\") pod \"payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67\" (UID: \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:06.624565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.624501    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67\" (UID: \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:06.638215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.625759    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67\" (UID: \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:06.667212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.665544    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8aba258-c5a6-4e67-9c6f-0037d6a61193-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j\" (UID: \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\") " pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:06.720439 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.715776    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjlw\" (UniqueName: \"kubernetes.io/projected/f8aba258-c5a6-4e67-9c6f-0037d6a61193-kube-api-access-xjjlw\") pod \"payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j\" (UID: \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\") " pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:06.731107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.728672    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:06.737853 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.737241    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kpq\" (UniqueName: \"kubernetes.io/projected/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-kube-api-access-d5kpq\") pod \"payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67\" (UID: \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:06.799647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:06.799570    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:07.098311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.098274    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"]
Oct 30 16:44:07.110071 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:44:07.110015    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aba258_c5a6_4e67_9c6f_0037d6a61193.slice/crio-068f1a4f9c004f376f9a8dcfcf7f3c9cd508aaeb43856d97234bcc6983601f8c WatchSource:0}: Error finding container 068f1a4f9c004f376f9a8dcfcf7f3c9cd508aaeb43856d97234bcc6983601f8c: Status 404 returned error can't find the container with id 068f1a4f9c004f376f9a8dcfcf7f3c9cd508aaeb43856d97234bcc6983601f8c
Oct 30 16:44:07.310760 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.310517    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" event={"ID":"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce","Type":"ContainerStarted","Data":"b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3"}
Oct 30 16:44:07.313155 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.313081    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j" event={"ID":"f8aba258-c5a6-4e67-9c6f-0037d6a61193","Type":"ContainerStarted","Data":"089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e"}
Oct 30 16:44:07.313155 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.313121    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j" event={"ID":"f8aba258-c5a6-4e67-9c6f-0037d6a61193","Type":"ContainerStarted","Data":"068f1a4f9c004f376f9a8dcfcf7f3c9cd508aaeb43856d97234bcc6983601f8c"}
Oct 30 16:44:07.313354 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.313338    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:07.341288 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.341264    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"]
Oct 30 16:44:07.344171 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:44:07.344146    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0ff4f5_ff1a_48f8_8746_3a4b49d4e5c3.slice/crio-ee4a234dc65533f790d7413b85acf6609ace0456cca01367f4eab90cca60212f WatchSource:0}: Error finding container ee4a234dc65533f790d7413b85acf6609ace0456cca01367f4eab90cca60212f: Status 404 returned error can't find the container with id ee4a234dc65533f790d7413b85acf6609ace0456cca01367f4eab90cca60212f
Oct 30 16:44:07.857792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.857756    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:44:07.858007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.857990    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 16:44:07.858075 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:07.858041    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 16:44:08.308865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.308831    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3099ed01-f441-4de2-bb5b-cbc6d167c809" path="/var/lib/kubelet/pods/3099ed01-f441-4de2-bb5b-cbc6d167c809/volumes"
Oct 30 16:44:08.322022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.321988    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67" event={"ID":"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3","Type":"ContainerStarted","Data":"adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2"}
Oct 30 16:44:08.322241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.322046    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67" event={"ID":"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3","Type":"ContainerStarted","Data":"ee4a234dc65533f790d7413b85acf6609ace0456cca01367f4eab90cca60212f"}
Oct 30 16:44:08.322428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.322411    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:08.362648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.362600    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j" podStartSLOduration=2.362584773 podStartE2EDuration="2.362584773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:07.401300154 +0000 UTC m=+1277.255178422" watchObservedRunningTime="2026-10-30 16:44:08.362584773 +0000 UTC m=+1278.216463040"
Oct 30 16:44:08.364754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.364715    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67" podStartSLOduration=2.36470222 podStartE2EDuration="2.36470222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:08.36041918 +0000 UTC m=+1278.214297446" watchObservedRunningTime="2026-10-30 16:44:08.36470222 +0000 UTC m=+1278.218580484"
Oct 30 16:44:08.663427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.663345    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"]
Oct 30 16:44:08.721920 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.721886    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"]
Oct 30 16:44:08.866716 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.866662    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 16:44:08.901209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:08.882851    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 16:44:09.857740 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:09.857704    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 16:44:09.858290 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:09.857767    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 16:44:10.857872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:10.857833    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 16:44:10.858346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:10.857898    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 16:44:11.857374 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:11.857338    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 16:44:11.857568 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:11.857391    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 16:44:12.647967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:12.647934    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"]
Oct 30 16:44:12.695201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:12.695138    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"]
Oct 30 16:44:13.254884 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.254852    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:44:13.255552 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.255493    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:44:13.260086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.260061    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:44:13.664276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.664162    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g" podUID="bc3f14dc-d400-4cb8-8460-112db7bc0abe" containerName="payload-processing" containerID="cri-o://631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b" gracePeriod=25
Oct 30 16:44:13.724514 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.724297    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk" podUID="a6ee8c42-1e5a-4c98-a798-0f3d986e04e3" containerName="payload-pre-processing" containerID="cri-o://1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e" gracePeriod=25
Oct 30 16:44:13.962882 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.962857    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:44:13.994052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.994013    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3f14dc-d400-4cb8-8460-112db7bc0abe-kube-api-access-j2xqx\" (UniqueName: \"kubernetes.io/projected/bc3f14dc-d400-4cb8-8460-112db7bc0abe-kube-api-access-j2xqx\") pod \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\" (UID: \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\") "
Oct 30 16:44:13.994248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.994138    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3f14dc-d400-4cb8-8460-112db7bc0abe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3f14dc-d400-4cb8-8460-112db7bc0abe-plugins-config-volume\") pod \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\" (UID: \"bc3f14dc-d400-4cb8-8460-112db7bc0abe\") "
Oct 30 16:44:13.994604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.994579    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3f14dc-d400-4cb8-8460-112db7bc0abe-plugins-config-volume" pod "bc3f14dc-d400-4cb8-8460-112db7bc0abe" (UID: "bc3f14dc-d400-4cb8-8460-112db7bc0abe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:13.994872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.994856    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3f14dc-d400-4cb8-8460-112db7bc0abe-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:13.997034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:13.997009    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3f14dc-d400-4cb8-8460-112db7bc0abe-kube-api-access-j2xqx" pod "bc3f14dc-d400-4cb8-8460-112db7bc0abe" (UID: "bc3f14dc-d400-4cb8-8460-112db7bc0abe"). InnerVolumeSpecName "kube-api-access-j2xqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:14.041695 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.041665    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:44:14.095997 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.095966    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-plugins-config-volume\") pod \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\" (UID: \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\") "
Oct 30 16:44:14.096264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.096242    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-kube-api-access-plcwl\" (UniqueName: \"kubernetes.io/projected/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-kube-api-access-plcwl\") pod \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\" (UID: \"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3\") "
Oct 30 16:44:14.096516 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.096455    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-plugins-config-volume" pod "a6ee8c42-1e5a-4c98-a798-0f3d986e04e3" (UID: "a6ee8c42-1e5a-4c98-a798-0f3d986e04e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:14.096807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.096789    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2xqx\" (UniqueName: \"kubernetes.io/projected/bc3f14dc-d400-4cb8-8460-112db7bc0abe-kube-api-access-j2xqx\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:14.096892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.096813    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:14.098885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.098854    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-kube-api-access-plcwl" pod "a6ee8c42-1e5a-4c98-a798-0f3d986e04e3" (UID: "a6ee8c42-1e5a-4c98-a798-0f3d986e04e3"). InnerVolumeSpecName "kube-api-access-plcwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:14.197774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.197687    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plcwl\" (UniqueName: \"kubernetes.io/projected/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3-kube-api-access-plcwl\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:14.391462 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.391429    4497 generic.go:356] "Generic (PLEG): container finished" podID="bc3f14dc-d400-4cb8-8460-112db7bc0abe" containerID="631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b" exitCode=0
Oct 30 16:44:14.391665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.391483    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"
Oct 30 16:44:14.391665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.391521    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g" event={"ID":"bc3f14dc-d400-4cb8-8460-112db7bc0abe","Type":"ContainerDied","Data":"631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b"}
Oct 30 16:44:14.391665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.391569    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g" event={"ID":"bc3f14dc-d400-4cb8-8460-112db7bc0abe","Type":"ContainerDied","Data":"8e8979a79084bef92f3e514373858385aff6abb0b97f51243013d8e9ee1d984b"}
Oct 30 16:44:14.391665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.391596    4497 scope.go:122] "RemoveContainer" containerID="631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b"
Oct 30 16:44:14.393108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.393084    4497 generic.go:356] "Generic (PLEG): container finished" podID="a6ee8c42-1e5a-4c98-a798-0f3d986e04e3" containerID="1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e" exitCode=0
Oct 30 16:44:14.393264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.393156    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"
Oct 30 16:44:14.393264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.393197    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk" event={"ID":"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3","Type":"ContainerDied","Data":"1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e"}
Oct 30 16:44:14.393264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.393243    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk" event={"ID":"a6ee8c42-1e5a-4c98-a798-0f3d986e04e3","Type":"ContainerDied","Data":"fae2e7cb439de23c06d0751c9955777608121b2a457fb56a8e7a9bb562c2b6c4"}
Oct 30 16:44:14.401701 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.401684    4497 scope.go:122] "RemoveContainer" containerID="631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b"
Oct 30 16:44:14.401985 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:14.401968    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b\": container with ID starting with 631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b not found: ID does not exist" containerID="631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b"
Oct 30 16:44:14.402051 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.401990    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b"} err="failed to get container status \"631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b\": rpc error: code = NotFound desc = could not find container \"631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b\": container with ID starting with 631dffc2f3b085295bd682629fac823253b4bc9e69c29f7a7dac0152d5b6893b not found: ID does not exist"
Oct 30 16:44:14.402051 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.402007    4497 scope.go:122] "RemoveContainer" containerID="1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e"
Oct 30 16:44:14.410687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.410667    4497 scope.go:122] "RemoveContainer" containerID="1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e"
Oct 30 16:44:14.410980 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:14.410961    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e\": container with ID starting with 1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e not found: ID does not exist" containerID="1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e"
Oct 30 16:44:14.411034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.410988    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e"} err="failed to get container status \"1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e\": rpc error: code = NotFound desc = could not find container \"1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e\": container with ID starting with 1a3f569d6acda74e484aad57bcae59a8ac64604ecd217faa04c8403c2178724e not found: ID does not exist"
Oct 30 16:44:14.437961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.437934    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"]
Oct 30 16:44:14.447974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.447901    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-5db7f8df78-2pp7g"]
Oct 30 16:44:14.472182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.472149    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"]
Oct 30 16:44:14.475511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:14.475488    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-59f749c667-m6jzk"]
Oct 30 16:44:15.267835 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.267803    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"]
Oct 30 16:44:15.268626 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.268604    4497 cpu_manager.go:414] "containerMap: removing container" podUID="bc3f14dc-d400-4cb8-8460-112db7bc0abe" containerName="payload-processing"
Oct 30 16:44:15.268697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.268629    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3f14dc-d400-4cb8-8460-112db7bc0abe" containerName="payload-processing"
Oct 30 16:44:15.268697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.268673    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a6ee8c42-1e5a-4c98-a798-0f3d986e04e3" containerName="payload-pre-processing"
Oct 30 16:44:15.268697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.268681    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ee8c42-1e5a-4c98-a798-0f3d986e04e3" containerName="payload-pre-processing"
Oct 30 16:44:15.268842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.268821    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ee8c42-1e5a-4c98-a798-0f3d986e04e3" containerName="payload-pre-processing"
Oct 30 16:44:15.268907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.268848    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3f14dc-d400-4cb8-8460-112db7bc0abe" containerName="payload-processing"
Oct 30 16:44:15.272072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.272053    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:15.287511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.287377    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4a4c68\""
Oct 30 16:44:15.287511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.287452    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4a4c68-dockercfg-vjkmk\""
Oct 30 16:44:15.294204 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.291158    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"]
Oct 30 16:44:15.315095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.315062    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08ba398-0330-4e93-8187-0ebf683dc069-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9\" (UID: \"e08ba398-0330-4e93-8187-0ebf683dc069\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:15.315315 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.315254    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrjw\" (UniqueName: \"kubernetes.io/projected/e08ba398-0330-4e93-8187-0ebf683dc069-kube-api-access-cdrjw\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9\" (UID: \"e08ba398-0330-4e93-8187-0ebf683dc069\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:15.420208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.419369    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08ba398-0330-4e93-8187-0ebf683dc069-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9\" (UID: \"e08ba398-0330-4e93-8187-0ebf683dc069\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:15.420208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.419606    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrjw\" (UniqueName: \"kubernetes.io/projected/e08ba398-0330-4e93-8187-0ebf683dc069-kube-api-access-cdrjw\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9\" (UID: \"e08ba398-0330-4e93-8187-0ebf683dc069\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:15.427207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.424216    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08ba398-0330-4e93-8187-0ebf683dc069-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9\" (UID: \"e08ba398-0330-4e93-8187-0ebf683dc069\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:15.494096 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.492239    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrjw\" (UniqueName: \"kubernetes.io/projected/e08ba398-0330-4e93-8187-0ebf683dc069-kube-api-access-cdrjw\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9\" (UID: \"e08ba398-0330-4e93-8187-0ebf683dc069\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:15.502900 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.501265    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"]
Oct 30 16:44:15.514450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.514415    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"]
Oct 30 16:44:15.514642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.514558    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:15.523257 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.522950    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75f25\" (UniqueName: \"kubernetes.io/projected/5cb5996c-426c-48d9-8623-f6cd46b90b7f-kube-api-access-75f25\") pod \"payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx\" (UID: \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:15.523485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.523464    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5996c-426c-48d9-8623-f6cd46b90b7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx\" (UID: \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:15.584960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.584928    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:15.625201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.625144    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5996c-426c-48d9-8623-f6cd46b90b7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx\" (UID: \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:15.625360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.625306    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75f25\" (UniqueName: \"kubernetes.io/projected/5cb5996c-426c-48d9-8623-f6cd46b90b7f-kube-api-access-75f25\") pod \"payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx\" (UID: \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:15.625930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.625903    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5996c-426c-48d9-8623-f6cd46b90b7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx\" (UID: \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:15.647571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.647538    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75f25\" (UniqueName: \"kubernetes.io/projected/5cb5996c-426c-48d9-8623-f6cd46b90b7f-kube-api-access-75f25\") pod \"payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx\" (UID: \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:15.833088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.832997    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:15.921037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:15.921000    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"]
Oct 30 16:44:15.923485 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:44:15.923459    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08ba398_0330_4e93_8187_0ebf683dc069.slice/crio-d4057c8d27265d555e14fda4df4caf5e25bb94c3eac6fe15bb47c059d308ea9a WatchSource:0}: Error finding container d4057c8d27265d555e14fda4df4caf5e25bb94c3eac6fe15bb47c059d308ea9a: Status 404 returned error can't find the container with id d4057c8d27265d555e14fda4df4caf5e25bb94c3eac6fe15bb47c059d308ea9a
Oct 30 16:44:16.137931 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.137734    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"]
Oct 30 16:44:16.141090 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:44:16.141059    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb5996c_426c_48d9_8623_f6cd46b90b7f.slice/crio-1c9d9340c6cd0aa8794d6abf723ae7b3a34866e85c45bb376e705a7ee945479e WatchSource:0}: Error finding container 1c9d9340c6cd0aa8794d6abf723ae7b3a34866e85c45bb376e705a7ee945479e: Status 404 returned error can't find the container with id 1c9d9340c6cd0aa8794d6abf723ae7b3a34866e85c45bb376e705a7ee945479e
Oct 30 16:44:16.300489 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.300457    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ee8c42-1e5a-4c98-a798-0f3d986e04e3" path="/var/lib/kubelet/pods/a6ee8c42-1e5a-4c98-a798-0f3d986e04e3/volumes"
Oct 30 16:44:16.300916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.300901    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3f14dc-d400-4cb8-8460-112db7bc0abe" path="/var/lib/kubelet/pods/bc3f14dc-d400-4cb8-8460-112db7bc0abe/volumes"
Oct 30 16:44:16.497177 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.497083    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx" event={"ID":"5cb5996c-426c-48d9-8623-f6cd46b90b7f","Type":"ContainerStarted","Data":"f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de"}
Oct 30 16:44:16.497177 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.497146    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx" event={"ID":"5cb5996c-426c-48d9-8623-f6cd46b90b7f","Type":"ContainerStarted","Data":"1c9d9340c6cd0aa8794d6abf723ae7b3a34866e85c45bb376e705a7ee945479e"}
Oct 30 16:44:16.497177 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.497174    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:16.498743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.498715    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9" event={"ID":"e08ba398-0330-4e93-8187-0ebf683dc069","Type":"ContainerStarted","Data":"406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5"}
Oct 30 16:44:16.498863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.498757    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9" event={"ID":"e08ba398-0330-4e93-8187-0ebf683dc069","Type":"ContainerStarted","Data":"d4057c8d27265d555e14fda4df4caf5e25bb94c3eac6fe15bb47c059d308ea9a"}
Oct 30 16:44:16.498863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.498777    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:16.596236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.593158    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx" podStartSLOduration=1.593137384 podStartE2EDuration="1.593137384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:16.543771418 +0000 UTC m=+1286.397649687" watchObservedRunningTime="2026-10-30 16:44:16.593137384 +0000 UTC m=+1286.447015649"
Oct 30 16:44:16.617261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:16.617199    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9" podStartSLOduration=1.617169147 podStartE2EDuration="1.617169147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:16.615969464 +0000 UTC m=+1286.469847729" watchObservedRunningTime="2026-10-30 16:44:16.617169147 +0000 UTC m=+1286.471047411"
Oct 30 16:44:17.648262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:17.648224    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558" podUID="ac197c92-563f-418c-9dd7-0275e17a254b" containerName="payload-processing" containerID="cri-o://f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668" gracePeriod=25
Oct 30 16:44:17.696058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:17.696007    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm" podUID="289679a3-d78c-45c5-90dd-2a36bfdf19de" containerName="payload-pre-processing" containerID="cri-o://a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f" gracePeriod=25
Oct 30 16:44:17.975774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:17.975745    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:44:18.028067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.028039    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:44:18.077035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.076999    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac197c92-563f-418c-9dd7-0275e17a254b-kube-api-access-gvsbf\" (UniqueName: \"kubernetes.io/projected/ac197c92-563f-418c-9dd7-0275e17a254b-kube-api-access-gvsbf\") pod \"ac197c92-563f-418c-9dd7-0275e17a254b\" (UID: \"ac197c92-563f-418c-9dd7-0275e17a254b\") "
Oct 30 16:44:18.077214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.077085    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289679a3-d78c-45c5-90dd-2a36bfdf19de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289679a3-d78c-45c5-90dd-2a36bfdf19de-plugins-config-volume\") pod \"289679a3-d78c-45c5-90dd-2a36bfdf19de\" (UID: \"289679a3-d78c-45c5-90dd-2a36bfdf19de\") "
Oct 30 16:44:18.077214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.077120    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac197c92-563f-418c-9dd7-0275e17a254b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac197c92-563f-418c-9dd7-0275e17a254b-plugins-config-volume\") pod \"ac197c92-563f-418c-9dd7-0275e17a254b\" (UID: \"ac197c92-563f-418c-9dd7-0275e17a254b\") "
Oct 30 16:44:18.077214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.077201    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289679a3-d78c-45c5-90dd-2a36bfdf19de-kube-api-access-6hbb2\" (UniqueName: \"kubernetes.io/projected/289679a3-d78c-45c5-90dd-2a36bfdf19de-kube-api-access-6hbb2\") pod \"289679a3-d78c-45c5-90dd-2a36bfdf19de\" (UID: \"289679a3-d78c-45c5-90dd-2a36bfdf19de\") "
Oct 30 16:44:18.077539 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.077511    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289679a3-d78c-45c5-90dd-2a36bfdf19de-plugins-config-volume" pod "289679a3-d78c-45c5-90dd-2a36bfdf19de" (UID: "289679a3-d78c-45c5-90dd-2a36bfdf19de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:18.077602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.077511    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac197c92-563f-418c-9dd7-0275e17a254b-plugins-config-volume" pod "ac197c92-563f-418c-9dd7-0275e17a254b" (UID: "ac197c92-563f-418c-9dd7-0275e17a254b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:18.078022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.078002    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289679a3-d78c-45c5-90dd-2a36bfdf19de-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:18.078022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.078022    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac197c92-563f-418c-9dd7-0275e17a254b-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:18.079147 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.079116    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac197c92-563f-418c-9dd7-0275e17a254b-kube-api-access-gvsbf" pod "ac197c92-563f-418c-9dd7-0275e17a254b" (UID: "ac197c92-563f-418c-9dd7-0275e17a254b"). InnerVolumeSpecName "kube-api-access-gvsbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:18.079608 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.079584    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289679a3-d78c-45c5-90dd-2a36bfdf19de-kube-api-access-6hbb2" pod "289679a3-d78c-45c5-90dd-2a36bfdf19de" (UID: "289679a3-d78c-45c5-90dd-2a36bfdf19de"). InnerVolumeSpecName "kube-api-access-6hbb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:18.179812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.179731    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvsbf\" (UniqueName: \"kubernetes.io/projected/ac197c92-563f-418c-9dd7-0275e17a254b-kube-api-access-gvsbf\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:18.179812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.179761    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hbb2\" (UniqueName: \"kubernetes.io/projected/289679a3-d78c-45c5-90dd-2a36bfdf19de-kube-api-access-6hbb2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:18.325713 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.325688    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:44:18.516519 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.516476    4497 generic.go:356] "Generic (PLEG): container finished" podID="289679a3-d78c-45c5-90dd-2a36bfdf19de" containerID="a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f" exitCode=0
Oct 30 16:44:18.516519 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.516521    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"
Oct 30 16:44:18.516723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.516565    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm" event={"ID":"289679a3-d78c-45c5-90dd-2a36bfdf19de","Type":"ContainerDied","Data":"a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f"}
Oct 30 16:44:18.516723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.516617    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm" event={"ID":"289679a3-d78c-45c5-90dd-2a36bfdf19de","Type":"ContainerDied","Data":"f32f0d2499b3e01f528dc6b5f613b788ae5a5ede54262766eb274ee5f942bc5d"}
Oct 30 16:44:18.516723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.516647    4497 scope.go:122] "RemoveContainer" containerID="a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f"
Oct 30 16:44:18.517990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.517969    4497 generic.go:356] "Generic (PLEG): container finished" podID="ac197c92-563f-418c-9dd7-0275e17a254b" containerID="f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668" exitCode=0
Oct 30 16:44:18.518071 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.518035    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"
Oct 30 16:44:18.518117 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.518059    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558" event={"ID":"ac197c92-563f-418c-9dd7-0275e17a254b","Type":"ContainerDied","Data":"f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668"}
Oct 30 16:44:18.518117 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.518109    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558" event={"ID":"ac197c92-563f-418c-9dd7-0275e17a254b","Type":"ContainerDied","Data":"44d92d06e839ed88da3003a4952c19bcc23e7ee2da7edda04ea02fbba6ea49b0"}
Oct 30 16:44:18.526382 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.526363    4497 scope.go:122] "RemoveContainer" containerID="a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f"
Oct 30 16:44:18.526648 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:18.526627    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f\": container with ID starting with a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f not found: ID does not exist" containerID="a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f"
Oct 30 16:44:18.526710 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.526656    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f"} err="failed to get container status \"a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f\": rpc error: code = NotFound desc = could not find container \"a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f\": container with ID starting with a350ba556b0c22e9a7caba7a4571f361f5c3bda082035b7b1c896cd11c018c1f not found: ID does not exist"
Oct 30 16:44:18.526757 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.526673    4497 scope.go:122] "RemoveContainer" containerID="f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668"
Oct 30 16:44:18.535938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.535917    4497 scope.go:122] "RemoveContainer" containerID="f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668"
Oct 30 16:44:18.536312 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:18.536286    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668\": container with ID starting with f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668 not found: ID does not exist" containerID="f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668"
Oct 30 16:44:18.536403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.536319    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668"} err="failed to get container status \"f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668\": rpc error: code = NotFound desc = could not find container \"f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668\": container with ID starting with f128f73397290c793423c71a725bb36e4c8fae5bfaf40a12f2548c904fd89668 not found: ID does not exist"
Oct 30 16:44:18.567484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.567302    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"]
Oct 30 16:44:18.580698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.580658    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-7bf6b5b65f-g5lzm"]
Oct 30 16:44:18.616875 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.602063    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"]
Oct 30 16:44:18.616875 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:18.606684    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-867cfd5bbb-6g558"]
Oct 30 16:44:19.332055 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:19.332029    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:44:19.996932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:19.996899    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"]
Oct 30 16:44:20.029789 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:20.029752    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"]
Oct 30 16:44:20.302657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:20.302624    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289679a3-d78c-45c5-90dd-2a36bfdf19de" path="/var/lib/kubelet/pods/289679a3-d78c-45c5-90dd-2a36bfdf19de/volumes"
Oct 30 16:44:20.303139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:20.303122    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac197c92-563f-418c-9dd7-0275e17a254b" path="/var/lib/kubelet/pods/ac197c92-563f-418c-9dd7-0275e17a254b/volumes"
Oct 30 16:44:21.919482 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:21.919445    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:44:23.367910 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.367875    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"]
Oct 30 16:44:23.368577 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.368559    4497 cpu_manager.go:414] "containerMap: removing container" podUID="289679a3-d78c-45c5-90dd-2a36bfdf19de" containerName="payload-pre-processing"
Oct 30 16:44:23.368641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.368578    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289679a3-d78c-45c5-90dd-2a36bfdf19de" containerName="payload-pre-processing"
Oct 30 16:44:23.368641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.368604    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ac197c92-563f-418c-9dd7-0275e17a254b" containerName="payload-processing"
Oct 30 16:44:23.368641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.368609    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac197c92-563f-418c-9dd7-0275e17a254b" containerName="payload-processing"
Oct 30 16:44:23.368755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.368718    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="289679a3-d78c-45c5-90dd-2a36bfdf19de" containerName="payload-pre-processing"
Oct 30 16:44:23.368755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.368731    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac197c92-563f-418c-9dd7-0275e17a254b" containerName="payload-processing"
Oct 30 16:44:23.372088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.372070    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.396133 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.396103    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-504442-serving-cert\""
Oct 30 16:44:23.399804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.399773    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"]
Oct 30 16:44:23.477955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.477913    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.478165 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.477978    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a6167e-9755-42ad-9d58-becd7a81cf2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.478329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.478306    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpcv\" (UniqueName: \"kubernetes.io/projected/a8a6167e-9755-42ad-9d58-becd7a81cf2c-kube-api-access-7vpcv\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.478419 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.478400    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.579621 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.579589    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.579791 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.579657    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a6167e-9755-42ad-9d58-becd7a81cf2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.579874 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.579824    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpcv\" (UniqueName: \"kubernetes.io/projected/a8a6167e-9755-42ad-9d58-becd7a81cf2c-kube-api-access-7vpcv\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.579930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.579890    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.580464 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.580442    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a6167e-9755-42ad-9d58-becd7a81cf2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.582157 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.582137    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.582372 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.582355    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.600870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.600842    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpcv\" (UniqueName: \"kubernetes.io/projected/a8a6167e-9755-42ad-9d58-becd7a81cf2c-kube-api-access-7vpcv\") pod \"maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.685757 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.685665    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:23.944027 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.943948    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"]
Oct 30 16:44:23.947865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.947848    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:23.964434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.964412    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 16:44:23.964548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.964412    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 16:44:23.970213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.970020    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 16:44:23.970213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.970045    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pmfs7\""
Oct 30 16:44:23.974518 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:23.974486    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"]
Oct 30 16:44:24.016320 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.016282    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"]
Oct 30 16:44:24.019847 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:44:24.019820    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a6167e_9755_42ad_9d58_becd7a81cf2c.slice/crio-7a037f8956fa84af8098bd2a7804bc5d0bcc665d061d92c727b612dbfb807542 WatchSource:0}: Error finding container 7a037f8956fa84af8098bd2a7804bc5d0bcc665d061d92c727b612dbfb807542: Status 404 returned error can't find the container with id 7a037f8956fa84af8098bd2a7804bc5d0bcc665d061d92c727b612dbfb807542
Oct 30 16:44:24.090696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.090661    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.090879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.090801    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.090879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.090836    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.090972 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.090893    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.091012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.090961    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.091082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.091065    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.091115 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.091096    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ll2\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-kube-api-access-m8ll2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.192182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.192144    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.192377 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.192223    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ll2\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-kube-api-access-m8ll2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.192425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.192403    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.192523 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.192507    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.192567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.192536    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.192615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.192599    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.192672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.192645    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.193000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.192967    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.194715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.194651    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.194945 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.194924    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.195011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.194955    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.195011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.194926    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.213196 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.213152    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.213530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.213509    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ll2\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-kube-api-access-m8ll2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bwmh4\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.261122 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.261083    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:24.563036 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.562987    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj" event={"ID":"a8a6167e-9755-42ad-9d58-becd7a81cf2c","Type":"ContainerStarted","Data":"3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3"}
Oct 30 16:44:24.563569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.563057    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj" event={"ID":"a8a6167e-9755-42ad-9d58-becd7a81cf2c","Type":"ContainerStarted","Data":"7a037f8956fa84af8098bd2a7804bc5d0bcc665d061d92c727b612dbfb807542"}
Oct 30 16:44:24.563569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.563201    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:24.586337 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.586313    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"]
Oct 30 16:44:24.588851 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:44:24.588826    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef2cc7e_99c1_4d82_bfdf_242fbd0440aa.slice/crio-5a8ed32710777d60922b5762be44d1fcdad59b99c3a0ec30a941e72743f36d18 WatchSource:0}: Error finding container 5a8ed32710777d60922b5762be44d1fcdad59b99c3a0ec30a941e72743f36d18: Status 404 returned error can't find the container with id 5a8ed32710777d60922b5762be44d1fcdad59b99c3a0ec30a941e72743f36d18
Oct 30 16:44:24.601837 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.601777    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj" podStartSLOduration=1.323151062 podStartE2EDuration="1.601764116s" totalImagesPullingTime="278.613054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:24.599833556 +0000 UTC m=+1294.453711823" watchObservedRunningTime="2026-10-30 16:44:24.601764116 +0000 UTC m=+1294.455642381"
Oct 30 16:44:24.997989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:24.997888    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw" podUID="3a1a1287-4249-465f-be17-795d3ee29d0b" containerName="payload-processing" containerID="cri-o://3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a" gracePeriod=25
Oct 30 16:44:25.030796 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.030737    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng" podUID="f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85" containerName="payload-pre-processing" containerID="cri-o://efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46" gracePeriod=25
Oct 30 16:44:25.326062 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.326036    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:44:25.350221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.350181    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:44:25.412403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.412364    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-kube-api-access-lx644\" (UniqueName: \"kubernetes.io/projected/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-kube-api-access-lx644\") pod \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\" (UID: \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\") "
Oct 30 16:44:25.412556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.412474    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1a1287-4249-465f-be17-795d3ee29d0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1a1287-4249-465f-be17-795d3ee29d0b-plugins-config-volume\") pod \"3a1a1287-4249-465f-be17-795d3ee29d0b\" (UID: \"3a1a1287-4249-465f-be17-795d3ee29d0b\") "
Oct 30 16:44:25.412690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.412671    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1a1287-4249-465f-be17-795d3ee29d0b-kube-api-access-kpqdf\" (UniqueName: \"kubernetes.io/projected/3a1a1287-4249-465f-be17-795d3ee29d0b-kube-api-access-kpqdf\") pod \"3a1a1287-4249-465f-be17-795d3ee29d0b\" (UID: \"3a1a1287-4249-465f-be17-795d3ee29d0b\") "
Oct 30 16:44:25.412795 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.412778    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-plugins-config-volume\") pod \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\" (UID: \"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85\") "
Oct 30 16:44:25.413101 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.413077    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1a1287-4249-465f-be17-795d3ee29d0b-plugins-config-volume" pod "3a1a1287-4249-465f-be17-795d3ee29d0b" (UID: "3a1a1287-4249-465f-be17-795d3ee29d0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:25.413419 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.413401    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1a1287-4249-465f-be17-795d3ee29d0b-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:25.413506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.413467    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-plugins-config-volume" pod "f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85" (UID: "f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:25.417009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.416974    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1a1287-4249-465f-be17-795d3ee29d0b-kube-api-access-kpqdf" pod "3a1a1287-4249-465f-be17-795d3ee29d0b" (UID: "3a1a1287-4249-465f-be17-795d3ee29d0b"). InnerVolumeSpecName "kube-api-access-kpqdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:25.420577 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.420550    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-kube-api-access-lx644" pod "f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85" (UID: "f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85"). InnerVolumeSpecName "kube-api-access-lx644". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:25.515133 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.515095    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpqdf\" (UniqueName: \"kubernetes.io/projected/3a1a1287-4249-465f-be17-795d3ee29d0b-kube-api-access-kpqdf\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:25.515133 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.515127    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:25.515133 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.515137    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx644\" (UniqueName: \"kubernetes.io/projected/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85-kube-api-access-lx644\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:25.572525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.572440    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4" event={"ID":"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa","Type":"ContainerStarted","Data":"5a8ed32710777d60922b5762be44d1fcdad59b99c3a0ec30a941e72743f36d18"}
Oct 30 16:44:25.574155 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.574121    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85" containerID="efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46" exitCode=0
Oct 30 16:44:25.574336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.574262    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng" event={"ID":"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85","Type":"ContainerDied","Data":"efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46"}
Oct 30 16:44:25.574336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.574304    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng" event={"ID":"f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85","Type":"ContainerDied","Data":"a5c3fe28f59a5bd1a026507b657d65dd57c47489fd97cb41c2e3b7876312e2b7"}
Oct 30 16:44:25.574336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.574326    4497 scope.go:122] "RemoveContainer" containerID="efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46"
Oct 30 16:44:25.574508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.574396    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"
Oct 30 16:44:25.578287 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.576110    4497 generic.go:356] "Generic (PLEG): container finished" podID="3a1a1287-4249-465f-be17-795d3ee29d0b" containerID="3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a" exitCode=0
Oct 30 16:44:25.578287 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.576170    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"
Oct 30 16:44:25.578287 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.576223    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw" event={"ID":"3a1a1287-4249-465f-be17-795d3ee29d0b","Type":"ContainerDied","Data":"3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a"}
Oct 30 16:44:25.578287 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.576273    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw" event={"ID":"3a1a1287-4249-465f-be17-795d3ee29d0b","Type":"ContainerDied","Data":"9d1718be62424cce3cf50ba54fa60affa04036ce004ae09139908b207baf4d4b"}
Oct 30 16:44:25.601974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.601945    4497 scope.go:122] "RemoveContainer" containerID="efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46"
Oct 30 16:44:25.608310 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:25.603155    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46\": container with ID starting with efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46 not found: ID does not exist" containerID="efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46"
Oct 30 16:44:25.608310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.603754    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46"} err="failed to get container status \"efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46\": rpc error: code = NotFound desc = could not find container \"efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46\": container with ID starting with efe13755dacf2b29f37b1be60b5e5cf9e34a72d01b9b30938ace8b64da4a1d46 not found: ID does not exist"
Oct 30 16:44:25.608310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.603781    4497 scope.go:122] "RemoveContainer" containerID="3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a"
Oct 30 16:44:25.638690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.638662    4497 scope.go:122] "RemoveContainer" containerID="3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a"
Oct 30 16:44:25.639352 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:25.639320    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a\": container with ID starting with 3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a not found: ID does not exist" containerID="3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a"
Oct 30 16:44:25.639480 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.639363    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a"} err="failed to get container status \"3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a\": rpc error: code = NotFound desc = could not find container \"3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a\": container with ID starting with 3daf92726b957c234286f387c0436ef8e84e28093457f3d54edb90eccaaef44a not found: ID does not exist"
Oct 30 16:44:25.668849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.668813    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"]
Oct 30 16:44:25.674200 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.674145    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-7dbfbfffdd-km6zw"]
Oct 30 16:44:25.705981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.705906    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"]
Oct 30 16:44:25.709892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:25.709856    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-5cfb648b79-tczng"]
Oct 30 16:44:26.303882 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:26.303845    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1a1287-4249-465f-be17-795d3ee29d0b" path="/var/lib/kubelet/pods/3a1a1287-4249-465f-be17-795d3ee29d0b/volumes"
Oct 30 16:44:26.304496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:26.304477    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85" path="/var/lib/kubelet/pods/f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85/volumes"
Oct 30 16:44:27.510484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:27.510453    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:44:27.510929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:27.510541    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:44:28.248979 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:28.248942    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:28.249071 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:28.249028    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:28.603560 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:28.603526    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4" event={"ID":"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa","Type":"ContainerStarted","Data":"1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370"}
Oct 30 16:44:28.604067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:28.603584    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:28.663933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:28.663850    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4" podStartSLOduration=2.007273153 podStartE2EDuration="5.663830328s" totalImagesPullingTime="3.656557175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:28.65383482 +0000 UTC m=+1298.507713086" watchObservedRunningTime="2026-10-30 16:44:28.663830328 +0000 UTC m=+1298.517708593"
Oct 30 16:44:29.396223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.396168    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"]
Oct 30 16:44:29.401574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.401523    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy" containerID="cri-o://18a0c549b8cbb596368af8e497ce4885177f08518dca75152857dcd376b345dc" gracePeriod=30
Oct 30 16:44:29.407107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.407083    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"]
Oct 30 16:44:29.407555 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.407506    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy" containerID="cri-o://b86c7e18af2c1607a2370478d78f5a1e17f3a2cf4cb8b37f6b03e7d16835de47" gracePeriod=30
Oct 30 16:44:29.424471 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.424431    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"]
Oct 30 16:44:29.425812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.425034    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy" containerID="cri-o://7d4186ee696250802deb67ec5236679259be096cc5e872dd47939e24b5bf3a01" gracePeriod=30
Oct 30 16:44:29.434249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.434215    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"]
Oct 30 16:44:29.435643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.434853    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy" containerID="cri-o://f2ad4f6d448ab390bcca0e3424a705885e2cb92a154e6c901fdbd0d842e5300b" gracePeriod=30
Oct 30 16:44:29.632242 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.631803    4497 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bwmh4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:44:29.632242 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:29.631870    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4" podUID="2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:44:30.581850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:30.581823    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:44:32.630750 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:32.630671    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:44:34.706134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.705886    4497 generic.go:356] "Generic (PLEG): container finished" podID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerID="7d4186ee696250802deb67ec5236679259be096cc5e872dd47939e24b5bf3a01" exitCode=0
Oct 30 16:44:34.706134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.706019    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" event={"ID":"d1db647c-9e15-447c-b59a-e9a40d2fa443","Type":"ContainerDied","Data":"7d4186ee696250802deb67ec5236679259be096cc5e872dd47939e24b5bf3a01"}
Oct 30 16:44:34.709283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.709030    4497 generic.go:356] "Generic (PLEG): container finished" podID="c0814617-855c-462f-86f0-bfecf957544d" containerID="18a0c549b8cbb596368af8e497ce4885177f08518dca75152857dcd376b345dc" exitCode=0
Oct 30 16:44:34.709283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.709198    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" event={"ID":"c0814617-855c-462f-86f0-bfecf957544d","Type":"ContainerDied","Data":"18a0c549b8cbb596368af8e497ce4885177f08518dca75152857dcd376b345dc"}
Oct 30 16:44:34.711709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.711683    4497 generic.go:356] "Generic (PLEG): container finished" podID="c59cf364-b247-4075-add6-def59435ea16" containerID="f2ad4f6d448ab390bcca0e3424a705885e2cb92a154e6c901fdbd0d842e5300b" exitCode=0
Oct 30 16:44:34.712106 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.711834    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" event={"ID":"c59cf364-b247-4075-add6-def59435ea16","Type":"ContainerDied","Data":"f2ad4f6d448ab390bcca0e3424a705885e2cb92a154e6c901fdbd0d842e5300b"}
Oct 30 16:44:34.714354 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.714148    4497 generic.go:356] "Generic (PLEG): container finished" podID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerID="b86c7e18af2c1607a2370478d78f5a1e17f3a2cf4cb8b37f6b03e7d16835de47" exitCode=0
Oct 30 16:44:34.714354 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.714197    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" event={"ID":"e635afca-a409-4a6f-8f3c-0a74609170fe","Type":"ContainerDied","Data":"b86c7e18af2c1607a2370478d78f5a1e17f3a2cf4cb8b37f6b03e7d16835de47"}
Oct 30 16:44:34.840961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.840930    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:44:34.854894 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.854869    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:44:34.869737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.869712    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:44:34.873200 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.873172    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:44:34.923618 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923579    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0814617-855c-462f-86f0-bfecf957544d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0814617-855c-462f-86f0-bfecf957544d-istiod-ca-cert\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.923807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923640    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-podinfo\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.923807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923668    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-podinfo\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.923807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923701    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-istio-token\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-istio-token\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.923807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923744    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-socket\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.923807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923793    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-data\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923832    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-credential-socket\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923868    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-certs\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923909    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-credential-socket\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923940    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-kube-api-access-ln8pt\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-kube-api-access-ln8pt\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.923973    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1db647c-9e15-447c-b59a-e9a40d2fa443-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1db647c-9e15-447c-b59a-e9a40d2fa443-istiod-ca-cert\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924019    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-credential-socket\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924077    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-data\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924077    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-data" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924088    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-socket" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.924110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924104    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-certs\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.924751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924140    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c59cf364-b247-4075-add6-def59435ea16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59cf364-b247-4075-add6-def59435ea16-istio-podinfo\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.924751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924524    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-socket\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.924751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924561    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-token\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-token\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.924751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924588    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-certs\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.924751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924623    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-credential-socket\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.924751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924651    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-kube-api-access-rg4vt\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-kube-api-access-rg4vt\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.924751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924722    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-data\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.925118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924760    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-certs\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.925118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924817    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-socket\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.925118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924849    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-envoy\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.925118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924886    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-envoy\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.925118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924930    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-kube-api-access-42z7l\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-kube-api-access-42z7l\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.925118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924987    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-data\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.925118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925045    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-socket\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.925118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925099    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-envoy\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.925536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925157    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-kube-api-access-9p2sg\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-kube-api-access-9p2sg\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.925536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925228    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0814617-855c-462f-86f0-bfecf957544d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0814617-855c-462f-86f0-bfecf957544d-istio-podinfo\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.925536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925262    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-token\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-token\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.925536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925361    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e635afca-a409-4a6f-8f3c-0a74609170fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e635afca-a409-4a6f-8f3c-0a74609170fe-istiod-ca-cert\") pod \"e635afca-a409-4a6f-8f3c-0a74609170fe\" (UID: \"e635afca-a409-4a6f-8f3c-0a74609170fe\") "
Oct 30 16:44:34.925536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925402    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-envoy\") pod \"d1db647c-9e15-447c-b59a-e9a40d2fa443\" (UID: \"d1db647c-9e15-447c-b59a-e9a40d2fa443\") "
Oct 30 16:44:34.925536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925455    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-istio-token\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-istio-token\") pod \"c0814617-855c-462f-86f0-bfecf957544d\" (UID: \"c0814617-855c-462f-86f0-bfecf957544d\") "
Oct 30 16:44:34.925536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925524    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59cf364-b247-4075-add6-def59435ea16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59cf364-b247-4075-add6-def59435ea16-istiod-ca-cert\") pod \"c59cf364-b247-4075-add6-def59435ea16\" (UID: \"c59cf364-b247-4075-add6-def59435ea16\") "
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924005    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0814617-855c-462f-86f0-bfecf957544d-istiod-ca-cert" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924280    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-certs" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924332    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-credential-socket" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924451    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-credential-socket" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924616    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1db647c-9e15-447c-b59a-e9a40d2fa443-istiod-ca-cert" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924687    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-certs" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924729    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-credential-socket" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924865    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-data" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924880    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-certs" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.924931    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-socket" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925159    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-credential-socket" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925786    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-certs" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.925940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.925934    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-data" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.926590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926110    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59cf364-b247-4075-add6-def59435ea16-istiod-ca-cert" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:34.926590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926575    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.926690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926597    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.926690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926612    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59cf364-b247-4075-add6-def59435ea16-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.926690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926626    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0814617-855c-462f-86f0-bfecf957544d-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.926690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926640    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.926690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926654    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.926690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926670    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.926690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926683    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.927009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926696    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.927009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926710    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1db647c-9e15-447c-b59a-e9a40d2fa443-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.927009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926723    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.927009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926737    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.927009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926751    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.927009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926763    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.927009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926775    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.927009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.926788    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:34.928638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.927409    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e635afca-a409-4a6f-8f3c-0a74609170fe-istiod-ca-cert" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:44:34.928638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.927550    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-podinfo" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:44:34.928638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.927640    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-istio-token" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:34.928638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.928126    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-socket" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.928638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.928396    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-data" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.929262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.929237    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-kube-api-access-rg4vt" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "kube-api-access-rg4vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:34.929415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.929312    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-token" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:34.929415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.929361    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0814617-855c-462f-86f0-bfecf957544d-istio-podinfo" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:44:34.929635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.929613    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-socket" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.932643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.932586    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-envoy" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.934468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.934432    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-istio-token" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:34.937113 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.936656    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-envoy" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.937113 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.937072    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-podinfo" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:44:34.937113 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.937086    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-envoy" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.937399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.937149    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-token" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:34.937399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.937153    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-envoy" pod "d1db647c-9e15-447c-b59a-e9a40d2fa443" (UID: "d1db647c-9e15-447c-b59a-e9a40d2fa443"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:44:34.937399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.937269    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c59cf364-b247-4075-add6-def59435ea16-istio-podinfo" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:44:34.937399 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.937326    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-kube-api-access-9p2sg" pod "c0814617-855c-462f-86f0-bfecf957544d" (UID: "c0814617-855c-462f-86f0-bfecf957544d"). InnerVolumeSpecName "kube-api-access-9p2sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:34.937603 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.937449    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-kube-api-access-ln8pt" pod "e635afca-a409-4a6f-8f3c-0a74609170fe" (UID: "e635afca-a409-4a6f-8f3c-0a74609170fe"). InnerVolumeSpecName "kube-api-access-ln8pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:34.938845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:34.938820    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-kube-api-access-42z7l" pod "c59cf364-b247-4075-add6-def59435ea16" (UID: "c59cf364-b247-4075-add6-def59435ea16"). InnerVolumeSpecName "kube-api-access-42z7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:44:35.028160 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028123    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p2sg\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-kube-api-access-9p2sg\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028160 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028159    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0814617-855c-462f-86f0-bfecf957544d-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028173    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028223    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e635afca-a409-4a6f-8f3c-0a74609170fe-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028236    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028248    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0814617-855c-462f-86f0-bfecf957544d-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028258    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1db647c-9e15-447c-b59a-e9a40d2fa443-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028270    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028280    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028292    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8pt\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-kube-api-access-ln8pt\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028305    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59cf364-b247-4075-add6-def59435ea16-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028317    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028329    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4vt\" (UniqueName: \"kubernetes.io/projected/d1db647c-9e15-447c-b59a-e9a40d2fa443-kube-api-access-rg4vt\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028340    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1db647c-9e15-447c-b59a-e9a40d2fa443-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028352    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0814617-855c-462f-86f0-bfecf957544d-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028363    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59cf364-b247-4075-add6-def59435ea16-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028374    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42z7l\" (UniqueName: \"kubernetes.io/projected/c59cf364-b247-4075-add6-def59435ea16-kube-api-access-42z7l\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028386    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028399    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.028444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.028412    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e635afca-a409-4a6f-8f3c-0a74609170fe-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:44:35.722839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.722755    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"
Oct 30 16:44:35.722839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.722773    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb" event={"ID":"c0814617-855c-462f-86f0-bfecf957544d","Type":"ContainerDied","Data":"b446365958c72f7d2a9be295d6b6f6b078c614dd9e5dbd4451335fc70400a514"}
Oct 30 16:44:35.722839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.722833    4497 scope.go:122] "RemoveContainer" containerID="18a0c549b8cbb596368af8e497ce4885177f08518dca75152857dcd376b345dc"
Oct 30 16:44:35.724452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.724432    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"
Oct 30 16:44:35.724541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.724467    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr" event={"ID":"c59cf364-b247-4075-add6-def59435ea16","Type":"ContainerDied","Data":"2dac896e67a5b33059e97774644680131787de1753ebf406357c06da7299c2a4"}
Oct 30 16:44:35.726181 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.726164    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"
Oct 30 16:44:35.726181 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.726169    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn" event={"ID":"e635afca-a409-4a6f-8f3c-0a74609170fe","Type":"ContainerDied","Data":"41ab70a25eb7622a1b9af9619fb1d949d928e1a1239e75fdbe8b41c5ff1f428e"}
Oct 30 16:44:35.743401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.741550    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w" event={"ID":"d1db647c-9e15-447c-b59a-e9a40d2fa443","Type":"ContainerDied","Data":"bb8c28180e3a389fcad10589ee87d4b298111f03012eac0689885e859a79c3ff"}
Oct 30 16:44:35.743401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.741585    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"
Oct 30 16:44:35.756382 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.756335    4497 scope.go:122] "RemoveContainer" containerID="f2ad4f6d448ab390bcca0e3424a705885e2cb92a154e6c901fdbd0d842e5300b"
Oct 30 16:44:35.767612 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.767592    4497 scope.go:122] "RemoveContainer" containerID="b86c7e18af2c1607a2370478d78f5a1e17f3a2cf4cb8b37f6b03e7d16835de47"
Oct 30 16:44:35.776951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.776931    4497 scope.go:122] "RemoveContainer" containerID="7d4186ee696250802deb67ec5236679259be096cc5e872dd47939e24b5bf3a01"
Oct 30 16:44:35.813997 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.813946    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"]
Oct 30 16:44:35.818733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.818702    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-zqqzn"]
Oct 30 16:44:35.852565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.852530    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"]
Oct 30 16:44:35.857618 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.857589    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-vdcbb"]
Oct 30 16:44:35.889451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.889415    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"]
Oct 30 16:44:35.893349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.893325    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6kt7w"]
Oct 30 16:44:35.921541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.921507    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"]
Oct 30 16:44:35.931742 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:35.931710    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-7gwmr"]
Oct 30 16:44:36.300382 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:36.300345    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0814617-855c-462f-86f0-bfecf957544d" path="/var/lib/kubelet/pods/c0814617-855c-462f-86f0-bfecf957544d/volumes"
Oct 30 16:44:36.301000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:36.300977    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59cf364-b247-4075-add6-def59435ea16" path="/var/lib/kubelet/pods/c59cf364-b247-4075-add6-def59435ea16/volumes"
Oct 30 16:44:36.301511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:36.301492    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" path="/var/lib/kubelet/pods/d1db647c-9e15-447c-b59a-e9a40d2fa443/volumes"
Oct 30 16:44:36.302329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:36.302311    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" path="/var/lib/kubelet/pods/e635afca-a409-4a6f-8f3c-0a74609170fe/volumes"
Oct 30 16:44:48.311346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:48.311309    4497 patch_prober.go:29] interesting pod/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:44:48.311763 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:48.311376    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:44:49.947037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:49.947000    4497 generic.go:356] "Generic (PLEG): container finished" podID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerID="b0e69a0db9afaaaff31d18bb4c1d07140cf7cc1cfdda866063e091568ee0ce1e" exitCode=0
Oct 30 16:44:49.947592 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:49.947072    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" event={"ID":"f4f1d7ae-9027-4ee7-8972-bc953e335c04","Type":"ContainerDied","Data":"b0e69a0db9afaaaff31d18bb4c1d07140cf7cc1cfdda866063e091568ee0ce1e"}
Oct 30 16:44:49.947690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:49.947655    4497 scope.go:122] "RemoveContainer" containerID="b0e69a0db9afaaaff31d18bb4c1d07140cf7cc1cfdda866063e091568ee0ce1e"
Oct 30 16:44:49.949564 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:49.949526    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:49.949653 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:49.949623    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:49.949693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:49.949672    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:44:50.954829 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:50.954794    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" event={"ID":"f4f1d7ae-9027-4ee7-8972-bc953e335c04","Type":"ContainerStarted","Data":"88367817e90f7f6ba6e7c443d1747a3819ae4a297c42ace13025269b2cbf3ade"}
Oct 30 16:44:51.664584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:51.664555    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:44:51.664769 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:51.664703    4497 patch_prober.go:29] interesting pod/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 16:44:51.664769 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:51.664743    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 16:44:51.940048 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:44:51.939965    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:44:52.664992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:52.664956    4497 patch_prober.go:29] interesting pod/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 16:44:52.665496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:52.665023    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 16:44:53.664872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:53.664837    4497 patch_prober.go:29] interesting pod/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 16:44:53.665058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:53.664895    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 16:44:54.664848 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:54.664810    4497 patch_prober.go:29] interesting pod/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 16:44:54.665118 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:54.664883    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 16:44:55.664837 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:55.664802    4497 patch_prober.go:29] interesting pod/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 16:44:55.665029 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:55.664855    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 16:44:56.668886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:56.668857    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:44:56.669395 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:56.669334    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:44:56.670103 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:44:56.670082    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:45:00.167301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167265    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"]
Oct 30 16:45:00.167862 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167850    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy"
Oct 30 16:45:00.167906 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167863    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy"
Oct 30 16:45:00.167906 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167882    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85" containerName="payload-pre-processing"
Oct 30 16:45:00.167906 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167888    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85" containerName="payload-pre-processing"
Oct 30 16:45:00.167906 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167906    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy"
Oct 30 16:45:00.168034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167910    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy"
Oct 30 16:45:00.168034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167917    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy"
Oct 30 16:45:00.168034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167921    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy"
Oct 30 16:45:00.168034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167936    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3a1a1287-4249-465f-be17-795d3ee29d0b" containerName="payload-processing"
Oct 30 16:45:00.168034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167940    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1a1287-4249-465f-be17-795d3ee29d0b" containerName="payload-processing"
Oct 30 16:45:00.168034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167948    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy"
Oct 30 16:45:00.168034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.167962    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy"
Oct 30 16:45:00.168267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.168058    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e635afca-a409-4a6f-8f3c-0a74609170fe" containerName="istio-proxy"
Oct 30 16:45:00.168267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.168068    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5947e42-d9ea-43c6-b3cf-0b5fd24f5e85" containerName="payload-pre-processing"
Oct 30 16:45:00.168267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.168077    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59cf364-b247-4075-add6-def59435ea16" containerName="istio-proxy"
Oct 30 16:45:00.168267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.168083    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1db647c-9e15-447c-b59a-e9a40d2fa443" containerName="istio-proxy"
Oct 30 16:45:00.168267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.168090    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1a1287-4249-465f-be17-795d3ee29d0b" containerName="payload-processing"
Oct 30 16:45:00.168267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.168097    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0814617-855c-462f-86f0-bfecf957544d" containerName="istio-proxy"
Oct 30 16:45:00.178872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.178842    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"]
Oct 30 16:45:00.179052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.178966    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:00.184300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.184270    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:00.193048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.193018    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"]
Oct 30 16:45:00.212355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.211825    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:00.235637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.235577    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"]
Oct 30 16:45:00.247765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.247689    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"]
Oct 30 16:45:00.252452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.252425    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"]
Oct 30 16:45:00.278562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.278534    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\""
Oct 30 16:45:00.331843 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.331806    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxcrd\" (UniqueName: \"kubernetes.io/projected/82453041-c363-4a8a-be3d-42553041ebb9-kube-api-access-pxcrd\") pod \"maas-api-key-cleanup-29889645-27r5k\" (UID: \"82453041-c363-4a8a-be3d-42553041ebb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:00.332277 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.332248    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82453041-c363-4a8a-be3d-42553041ebb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-27r5k\" (UID: \"82453041-c363-4a8a-be3d-42553041ebb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:00.332612 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.332592    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flpk\" (UniqueName: \"kubernetes.io/projected/7e4d51da-a8a2-4554-94fd-180a1704ba03-kube-api-access-6flpk\") pod \"maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss\" (UID: \"7e4d51da-a8a2-4554-94fd-180a1704ba03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:00.332754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.332738    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4d51da-a8a2-4554-94fd-180a1704ba03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss\" (UID: \"7e4d51da-a8a2-4554-94fd-180a1704ba03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:00.332911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.332889    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm57h\" (UniqueName: \"kubernetes.io/projected/180063f7-7cf4-4c66-8e60-2b40551028c2-kube-api-access-wm57h\") pod \"maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft\" (UID: \"180063f7-7cf4-4c66-8e60-2b40551028c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:00.333147 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.332996    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180063f7-7cf4-4c66-8e60-2b40551028c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft\" (UID: \"180063f7-7cf4-4c66-8e60-2b40551028c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:00.435009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.434924    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxcrd\" (UniqueName: \"kubernetes.io/projected/82453041-c363-4a8a-be3d-42553041ebb9-kube-api-access-pxcrd\") pod \"maas-api-key-cleanup-29889645-27r5k\" (UID: \"82453041-c363-4a8a-be3d-42553041ebb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:00.435009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.435003    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82453041-c363-4a8a-be3d-42553041ebb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-27r5k\" (UID: \"82453041-c363-4a8a-be3d-42553041ebb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:00.435322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.435041    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flpk\" (UniqueName: \"kubernetes.io/projected/7e4d51da-a8a2-4554-94fd-180a1704ba03-kube-api-access-6flpk\") pod \"maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss\" (UID: \"7e4d51da-a8a2-4554-94fd-180a1704ba03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:00.435322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.435057    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4d51da-a8a2-4554-94fd-180a1704ba03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss\" (UID: \"7e4d51da-a8a2-4554-94fd-180a1704ba03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:00.435322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.435076    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm57h\" (UniqueName: \"kubernetes.io/projected/180063f7-7cf4-4c66-8e60-2b40551028c2-kube-api-access-wm57h\") pod \"maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft\" (UID: \"180063f7-7cf4-4c66-8e60-2b40551028c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:00.435322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.435104    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180063f7-7cf4-4c66-8e60-2b40551028c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft\" (UID: \"180063f7-7cf4-4c66-8e60-2b40551028c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:00.435857 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.435836    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180063f7-7cf4-4c66-8e60-2b40551028c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft\" (UID: \"180063f7-7cf4-4c66-8e60-2b40551028c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:00.435947 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.435924    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82453041-c363-4a8a-be3d-42553041ebb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-27r5k\" (UID: \"82453041-c363-4a8a-be3d-42553041ebb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:00.435983 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.435924    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4d51da-a8a2-4554-94fd-180a1704ba03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss\" (UID: \"7e4d51da-a8a2-4554-94fd-180a1704ba03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:00.460247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.460094    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flpk\" (UniqueName: \"kubernetes.io/projected/7e4d51da-a8a2-4554-94fd-180a1704ba03-kube-api-access-6flpk\") pod \"maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss\" (UID: \"7e4d51da-a8a2-4554-94fd-180a1704ba03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:00.477035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.476966    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm57h\" (UniqueName: \"kubernetes.io/projected/180063f7-7cf4-4c66-8e60-2b40551028c2-kube-api-access-wm57h\") pod \"maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft\" (UID: \"180063f7-7cf4-4c66-8e60-2b40551028c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:00.482327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.482288    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxcrd\" (UniqueName: \"kubernetes.io/projected/82453041-c363-4a8a-be3d-42553041ebb9-kube-api-access-pxcrd\") pod \"maas-api-key-cleanup-29889645-27r5k\" (UID: \"82453041-c363-4a8a-be3d-42553041ebb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:00.511055 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.511019    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:00.522902 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.522874    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:00.548637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.548599    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:00.824334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.824300    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"]
Oct 30 16:45:00.832042 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:45:00.832009    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82453041_c363_4a8a_be3d_42553041ebb9.slice/crio-f1eb44018b70c3ebb24dc60462015956dd58f7588ed6f4ea097762d093d94167 WatchSource:0}: Error finding container f1eb44018b70c3ebb24dc60462015956dd58f7588ed6f4ea097762d093d94167: Status 404 returned error can't find the container with id f1eb44018b70c3ebb24dc60462015956dd58f7588ed6f4ea097762d093d94167
Oct 30 16:45:00.843963 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.843941    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"]
Oct 30 16:45:00.845743 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:45:00.845707    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4d51da_a8a2_4554_94fd_180a1704ba03.slice/crio-3fe54a2cfc4ec301db514df4097cf62f2aef625dd4d584df23588899cd20f855 WatchSource:0}: Error finding container 3fe54a2cfc4ec301db514df4097cf62f2aef625dd4d584df23588899cd20f855: Status 404 returned error can't find the container with id 3fe54a2cfc4ec301db514df4097cf62f2aef625dd4d584df23588899cd20f855
Oct 30 16:45:00.894409 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:00.894378    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"]
Oct 30 16:45:00.898156 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:45:00.898132    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180063f7_7cf4_4c66_8e60_2b40551028c2.slice/crio-3e07358fdc14e99b5cc715ba087957f4a54e6aa62573ce1c4496bf49f6862938 WatchSource:0}: Error finding container 3e07358fdc14e99b5cc715ba087957f4a54e6aa62573ce1c4496bf49f6862938: Status 404 returned error can't find the container with id 3e07358fdc14e99b5cc715ba087957f4a54e6aa62573ce1c4496bf49f6862938
Oct 30 16:45:01.052154 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:01.052120    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" event={"ID":"82453041-c363-4a8a-be3d-42553041ebb9","Type":"ContainerStarted","Data":"f1eb44018b70c3ebb24dc60462015956dd58f7588ed6f4ea097762d093d94167"}
Oct 30 16:45:01.053533 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:01.053502    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss" event={"ID":"7e4d51da-a8a2-4554-94fd-180a1704ba03","Type":"ContainerStarted","Data":"3fe54a2cfc4ec301db514df4097cf62f2aef625dd4d584df23588899cd20f855"}
Oct 30 16:45:01.054728 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:01.054709    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" event={"ID":"180063f7-7cf4-4c66-8e60-2b40551028c2","Type":"ContainerStarted","Data":"3e07358fdc14e99b5cc715ba087957f4a54e6aa62573ce1c4496bf49f6862938"}
Oct 30 16:45:02.063957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.063920    4497 generic.go:356] "Generic (PLEG): container finished" podID="82453041-c363-4a8a-be3d-42553041ebb9" containerID="c6dbaf19af65c3500c2209bd240ee22cb891ce9c1f2015109465640aad265018" exitCode=60
Oct 30 16:45:02.064487 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.064000    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" event={"ID":"82453041-c363-4a8a-be3d-42553041ebb9","Type":"ContainerDied","Data":"c6dbaf19af65c3500c2209bd240ee22cb891ce9c1f2015109465640aad265018"}
Oct 30 16:45:02.064558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.064487    4497 scope.go:122] "RemoveContainer" containerID="c6dbaf19af65c3500c2209bd240ee22cb891ce9c1f2015109465640aad265018"
Oct 30 16:45:02.065763 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.065741    4497 generic.go:356] "Generic (PLEG): container finished" podID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerID="19a19654368eb772bd43cbc170fa8164b13861bcf98d501201d94b3b04a5b163" exitCode=60
Oct 30 16:45:02.065891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.065869    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss" event={"ID":"7e4d51da-a8a2-4554-94fd-180a1704ba03","Type":"ContainerDied","Data":"19a19654368eb772bd43cbc170fa8164b13861bcf98d501201d94b3b04a5b163"}
Oct 30 16:45:02.066159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.066139    4497 scope.go:122] "RemoveContainer" containerID="19a19654368eb772bd43cbc170fa8164b13861bcf98d501201d94b3b04a5b163"
Oct 30 16:45:02.067516 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.067494    4497 generic.go:356] "Generic (PLEG): container finished" podID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerID="0db389cda34ba8fcf53a94e2d7813acc5b4f372be28d6ff7cfe393a316f97c24" exitCode=60
Oct 30 16:45:02.067627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.067585    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" event={"ID":"180063f7-7cf4-4c66-8e60-2b40551028c2","Type":"ContainerDied","Data":"0db389cda34ba8fcf53a94e2d7813acc5b4f372be28d6ff7cfe393a316f97c24"}
Oct 30 16:45:02.067792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:02.067778    4497 scope.go:122] "RemoveContainer" containerID="0db389cda34ba8fcf53a94e2d7813acc5b4f372be28d6ff7cfe393a316f97c24"
Oct 30 16:45:03.086843 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:03.086739    4497 generic.go:356] "Generic (PLEG): container finished" podID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerID="23d1a5ff2ce9ad3ce149ad2754d42bef2a72b71181a78472da69b63534ab1fb6" exitCode=60
Oct 30 16:45:03.087222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:03.086907    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss" event={"ID":"7e4d51da-a8a2-4554-94fd-180a1704ba03","Type":"ContainerDied","Data":"23d1a5ff2ce9ad3ce149ad2754d42bef2a72b71181a78472da69b63534ab1fb6"}
Oct 30 16:45:03.087222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:03.086964    4497 scope.go:122] "RemoveContainer" containerID="19a19654368eb772bd43cbc170fa8164b13861bcf98d501201d94b3b04a5b163"
Oct 30 16:45:03.087537 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:03.087517    4497 scope.go:122] "RemoveContainer" containerID="23d1a5ff2ce9ad3ce149ad2754d42bef2a72b71181a78472da69b63534ab1fb6"
Oct 30 16:45:03.087941 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:03.087914    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss_odh-ai-gateway-infra(7e4d51da-a8a2-4554-94fd-180a1704ba03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03"
Oct 30 16:45:03.103609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:03.103378    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" event={"ID":"180063f7-7cf4-4c66-8e60-2b40551028c2","Type":"ContainerStarted","Data":"196ca0d6b7dc31b4670f22f62390747dc6164496efb9cfe3cc3934d0d7b465e7"}
Oct 30 16:45:03.112067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:03.112026    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" event={"ID":"82453041-c363-4a8a-be3d-42553041ebb9","Type":"ContainerStarted","Data":"ece06aab8e4d0128fe2aaa75e82752e6ce7cb0583c59baac00e54a7cd5a96b55"}
Oct 30 16:45:03.167220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:03.167151    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" podStartSLOduration=3.167134429 podStartE2EDuration="3.167134429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.164092969 +0000 UTC m=+1333.017971231" watchObservedRunningTime="2026-10-30 16:45:03.167134429 +0000 UTC m=+1333.021012693"
Oct 30 16:45:03.201641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:03.201593    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" podStartSLOduration=3.201578091 podStartE2EDuration="3.201578091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.200421167 +0000 UTC m=+1333.054299433" watchObservedRunningTime="2026-10-30 16:45:03.201578091 +0000 UTC m=+1333.055456402"
Oct 30 16:45:04.136620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.136589    4497 scope.go:122] "RemoveContainer" containerID="23d1a5ff2ce9ad3ce149ad2754d42bef2a72b71181a78472da69b63534ab1fb6"
Oct 30 16:45:04.137425 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:04.137397    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss_odh-ai-gateway-infra(7e4d51da-a8a2-4554-94fd-180a1704ba03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03"
Oct 30 16:45:04.141090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.141051    4497 generic.go:356] "Generic (PLEG): container finished" podID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerID="196ca0d6b7dc31b4670f22f62390747dc6164496efb9cfe3cc3934d0d7b465e7" exitCode=60
Oct 30 16:45:04.141702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.141645    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" event={"ID":"180063f7-7cf4-4c66-8e60-2b40551028c2","Type":"ContainerDied","Data":"196ca0d6b7dc31b4670f22f62390747dc6164496efb9cfe3cc3934d0d7b465e7"}
Oct 30 16:45:04.141874 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.141724    4497 scope.go:122] "RemoveContainer" containerID="0db389cda34ba8fcf53a94e2d7813acc5b4f372be28d6ff7cfe393a316f97c24"
Oct 30 16:45:04.141973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.141882    4497 scope.go:122] "RemoveContainer" containerID="196ca0d6b7dc31b4670f22f62390747dc6164496efb9cfe3cc3934d0d7b465e7"
Oct 30 16:45:04.142337 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:04.142311    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft_odh-ai-gateway-infra(180063f7-7cf4-4c66-8e60-2b40551028c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2"
Oct 30 16:45:04.148432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.148405    4497 generic.go:356] "Generic (PLEG): container finished" podID="82453041-c363-4a8a-be3d-42553041ebb9" containerID="ece06aab8e4d0128fe2aaa75e82752e6ce7cb0583c59baac00e54a7cd5a96b55" exitCode=60
Oct 30 16:45:04.148576 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.148458    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" event={"ID":"82453041-c363-4a8a-be3d-42553041ebb9","Type":"ContainerDied","Data":"ece06aab8e4d0128fe2aaa75e82752e6ce7cb0583c59baac00e54a7cd5a96b55"}
Oct 30 16:45:04.148911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.148894    4497 scope.go:122] "RemoveContainer" containerID="ece06aab8e4d0128fe2aaa75e82752e6ce7cb0583c59baac00e54a7cd5a96b55"
Oct 30 16:45:04.149296 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:04.149222    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-27r5k_odh-ai-gateway-infra(82453041-c363-4a8a-be3d-42553041ebb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" podUID="82453041-c363-4a8a-be3d-42553041ebb9"
Oct 30 16:45:04.159711 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:04.159685    4497 scope.go:122] "RemoveContainer" containerID="c6dbaf19af65c3500c2209bd240ee22cb891ce9c1f2015109465640aad265018"
Oct 30 16:45:05.156985 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:05.156959    4497 scope.go:122] "RemoveContainer" containerID="196ca0d6b7dc31b4670f22f62390747dc6164496efb9cfe3cc3934d0d7b465e7"
Oct 30 16:45:05.157541 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:05.157345    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft_odh-ai-gateway-infra(180063f7-7cf4-4c66-8e60-2b40551028c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2"
Oct 30 16:45:05.158649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:05.158626    4497 scope.go:122] "RemoveContainer" containerID="ece06aab8e4d0128fe2aaa75e82752e6ce7cb0583c59baac00e54a7cd5a96b55"
Oct 30 16:45:05.158936 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:05.158916    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-27r5k_odh-ai-gateway-infra(82453041-c363-4a8a-be3d-42553041ebb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" podUID="82453041-c363-4a8a-be3d-42553041ebb9"
Oct 30 16:45:08.784660 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:08.784623    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"]
Oct 30 16:45:08.817751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:08.817714    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"]
Oct 30 16:45:13.292692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:13.292659    4497 scope.go:122] "RemoveContainer" containerID="ece06aab8e4d0128fe2aaa75e82752e6ce7cb0583c59baac00e54a7cd5a96b55"
Oct 30 16:45:13.785544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:13.785469    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx" podUID="5cb5996c-426c-48d9-8623-f6cd46b90b7f" containerName="payload-processing" containerID="cri-o://f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de" gracePeriod=25
Oct 30 16:45:13.818300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:13.818264    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9" podUID="e08ba398-0330-4e93-8187-0ebf683dc069" containerName="payload-pre-processing" containerID="cri-o://406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5" gracePeriod=25
Oct 30 16:45:14.109732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.109705    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:45:14.132073 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.132048    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:45:14.226220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.226172    4497 generic.go:356] "Generic (PLEG): container finished" podID="e08ba398-0330-4e93-8187-0ebf683dc069" containerID="406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5" exitCode=0
Oct 30 16:45:14.226401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.226219    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9" event={"ID":"e08ba398-0330-4e93-8187-0ebf683dc069","Type":"ContainerDied","Data":"406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5"}
Oct 30 16:45:14.226401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.226278    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9" event={"ID":"e08ba398-0330-4e93-8187-0ebf683dc069","Type":"ContainerDied","Data":"d4057c8d27265d555e14fda4df4caf5e25bb94c3eac6fe15bb47c059d308ea9a"}
Oct 30 16:45:14.226401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.226303    4497 scope.go:122] "RemoveContainer" containerID="406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5"
Oct 30 16:45:14.226401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.226234    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"
Oct 30 16:45:14.227723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.227700    4497 generic.go:356] "Generic (PLEG): container finished" podID="5cb5996c-426c-48d9-8623-f6cd46b90b7f" containerID="f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de" exitCode=0
Oct 30 16:45:14.227849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.227746    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"
Oct 30 16:45:14.227849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.227786    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx" event={"ID":"5cb5996c-426c-48d9-8623-f6cd46b90b7f","Type":"ContainerDied","Data":"f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de"}
Oct 30 16:45:14.227849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.227846    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx" event={"ID":"5cb5996c-426c-48d9-8623-f6cd46b90b7f","Type":"ContainerDied","Data":"1c9d9340c6cd0aa8794d6abf723ae7b3a34866e85c45bb376e705a7ee945479e"}
Oct 30 16:45:14.229711 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.229688    4497 generic.go:356] "Generic (PLEG): container finished" podID="82453041-c363-4a8a-be3d-42553041ebb9" containerID="6994ada03306dc4b835568c78607a4d145a24dc8b3b86e5682c070e377bf5b7d" exitCode=60
Oct 30 16:45:14.229830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.229727    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" event={"ID":"82453041-c363-4a8a-be3d-42553041ebb9","Type":"ContainerDied","Data":"6994ada03306dc4b835568c78607a4d145a24dc8b3b86e5682c070e377bf5b7d"}
Oct 30 16:45:14.230104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.230089    4497 scope.go:122] "RemoveContainer" containerID="6994ada03306dc4b835568c78607a4d145a24dc8b3b86e5682c070e377bf5b7d"
Oct 30 16:45:14.230406 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:14.230388    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-27r5k_odh-ai-gateway-infra(82453041-c363-4a8a-be3d-42553041ebb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" podUID="82453041-c363-4a8a-be3d-42553041ebb9"
Oct 30 16:45:14.236244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.236225    4497 scope.go:122] "RemoveContainer" containerID="406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5"
Oct 30 16:45:14.236514 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:14.236490    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5\": container with ID starting with 406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5 not found: ID does not exist" containerID="406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5"
Oct 30 16:45:14.236566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.236527    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5"} err="failed to get container status \"406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5\": rpc error: code = NotFound desc = could not find container \"406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5\": container with ID starting with 406463249b8ff3b538285a93dcb3a063b2ca2be0b1b6d40669abdbb29af2fdf5 not found: ID does not exist"
Oct 30 16:45:14.236566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.236551    4497 scope.go:122] "RemoveContainer" containerID="f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de"
Oct 30 16:45:14.245546 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.245525    4497 scope.go:122] "RemoveContainer" containerID="f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de"
Oct 30 16:45:14.245838 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:14.245819    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de\": container with ID starting with f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de not found: ID does not exist" containerID="f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de"
Oct 30 16:45:14.245919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.245849    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de"} err="failed to get container status \"f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de\": rpc error: code = NotFound desc = could not find container \"f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de\": container with ID starting with f260d615061d1bbc279d126218fc064c2a3d66399427ee9aa53cad0edde9d8de not found: ID does not exist"
Oct 30 16:45:14.245919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.245875    4497 scope.go:122] "RemoveContainer" containerID="ece06aab8e4d0128fe2aaa75e82752e6ce7cb0583c59baac00e54a7cd5a96b55"
Oct 30 16:45:14.307295 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.307064    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb5996c-426c-48d9-8623-f6cd46b90b7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5996c-426c-48d9-8623-f6cd46b90b7f-plugins-config-volume\") pod \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\" (UID: \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\") "
Oct 30 16:45:14.307687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.307348    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08ba398-0330-4e93-8187-0ebf683dc069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08ba398-0330-4e93-8187-0ebf683dc069-plugins-config-volume\") pod \"e08ba398-0330-4e93-8187-0ebf683dc069\" (UID: \"e08ba398-0330-4e93-8187-0ebf683dc069\") "
Oct 30 16:45:14.307687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.307458    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb5996c-426c-48d9-8623-f6cd46b90b7f-plugins-config-volume" pod "5cb5996c-426c-48d9-8623-f6cd46b90b7f" (UID: "5cb5996c-426c-48d9-8623-f6cd46b90b7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:45:14.307854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.307834    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08ba398-0330-4e93-8187-0ebf683dc069-plugins-config-volume" pod "e08ba398-0330-4e93-8187-0ebf683dc069" (UID: "e08ba398-0330-4e93-8187-0ebf683dc069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:45:14.307964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.307936    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb5996c-426c-48d9-8623-f6cd46b90b7f-kube-api-access-75f25\" (UniqueName: \"kubernetes.io/projected/5cb5996c-426c-48d9-8623-f6cd46b90b7f-kube-api-access-75f25\") pod \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\" (UID: \"5cb5996c-426c-48d9-8623-f6cd46b90b7f\") "
Oct 30 16:45:14.308321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.308298    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08ba398-0330-4e93-8187-0ebf683dc069-kube-api-access-cdrjw\" (UniqueName: \"kubernetes.io/projected/e08ba398-0330-4e93-8187-0ebf683dc069-kube-api-access-cdrjw\") pod \"e08ba398-0330-4e93-8187-0ebf683dc069\" (UID: \"e08ba398-0330-4e93-8187-0ebf683dc069\") "
Oct 30 16:45:14.308999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.308977    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5996c-426c-48d9-8623-f6cd46b90b7f-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:14.309101 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.309003    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08ba398-0330-4e93-8187-0ebf683dc069-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:14.310998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.310973    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08ba398-0330-4e93-8187-0ebf683dc069-kube-api-access-cdrjw" pod "e08ba398-0330-4e93-8187-0ebf683dc069" (UID: "e08ba398-0330-4e93-8187-0ebf683dc069"). InnerVolumeSpecName "kube-api-access-cdrjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:45:14.311104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.311045    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb5996c-426c-48d9-8623-f6cd46b90b7f-kube-api-access-75f25" pod "5cb5996c-426c-48d9-8623-f6cd46b90b7f" (UID: "5cb5996c-426c-48d9-8623-f6cd46b90b7f"). InnerVolumeSpecName "kube-api-access-75f25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:45:14.410450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.410371    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75f25\" (UniqueName: \"kubernetes.io/projected/5cb5996c-426c-48d9-8623-f6cd46b90b7f-kube-api-access-75f25\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:14.410450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.410399    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrjw\" (UniqueName: \"kubernetes.io/projected/e08ba398-0330-4e93-8187-0ebf683dc069-kube-api-access-cdrjw\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:14.563581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.563548    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"]
Oct 30 16:45:14.572680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.572648    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-bcc9b55cd-jmdmx"]
Oct 30 16:45:14.601750 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.601720    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"]
Oct 30 16:45:14.607243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:14.607214    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-556b9885f9-dsqs9"]
Oct 30 16:45:15.285121 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:15.285086    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"]
Oct 30 16:45:15.474480 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:15.474456    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:15.524726 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:15.524690    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82453041-c363-4a8a-be3d-42553041ebb9-kube-api-access-pxcrd\" (UniqueName: \"kubernetes.io/projected/82453041-c363-4a8a-be3d-42553041ebb9-kube-api-access-pxcrd\") pod \"82453041-c363-4a8a-be3d-42553041ebb9\" (UID: \"82453041-c363-4a8a-be3d-42553041ebb9\") "
Oct 30 16:45:15.524886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:15.524775    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82453041-c363-4a8a-be3d-42553041ebb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82453041-c363-4a8a-be3d-42553041ebb9-maas-api-service-ca\") pod \"82453041-c363-4a8a-be3d-42553041ebb9\" (UID: \"82453041-c363-4a8a-be3d-42553041ebb9\") "
Oct 30 16:45:15.525259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:15.525230    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82453041-c363-4a8a-be3d-42553041ebb9-maas-api-service-ca" pod "82453041-c363-4a8a-be3d-42553041ebb9" (UID: "82453041-c363-4a8a-be3d-42553041ebb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:45:15.525499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:15.525474    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82453041-c363-4a8a-be3d-42553041ebb9-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:15.526950 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:15.526921    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82453041-c363-4a8a-be3d-42553041ebb9-kube-api-access-pxcrd" pod "82453041-c363-4a8a-be3d-42553041ebb9" (UID: "82453041-c363-4a8a-be3d-42553041ebb9"). InnerVolumeSpecName "kube-api-access-pxcrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:45:15.626407 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:15.626366    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxcrd\" (UniqueName: \"kubernetes.io/projected/82453041-c363-4a8a-be3d-42553041ebb9-kube-api-access-pxcrd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:16.248987 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:16.248958    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"
Oct 30 16:45:16.249162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:16.248985    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k" event={"ID":"82453041-c363-4a8a-be3d-42553041ebb9","Type":"ContainerDied","Data":"f1eb44018b70c3ebb24dc60462015956dd58f7588ed6f4ea097762d093d94167"}
Oct 30 16:45:16.249162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:16.249042    4497 scope.go:122] "RemoveContainer" containerID="6994ada03306dc4b835568c78607a4d145a24dc8b3b86e5682c070e377bf5b7d"
Oct 30 16:45:16.293502 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:16.293472    4497 scope.go:122] "RemoveContainer" containerID="23d1a5ff2ce9ad3ce149ad2754d42bef2a72b71181a78472da69b63534ab1fb6"
Oct 30 16:45:16.301834 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:16.301798    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb5996c-426c-48d9-8623-f6cd46b90b7f" path="/var/lib/kubelet/pods/5cb5996c-426c-48d9-8623-f6cd46b90b7f/volumes"
Oct 30 16:45:16.302227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:16.302214    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08ba398-0330-4e93-8187-0ebf683dc069" path="/var/lib/kubelet/pods/e08ba398-0330-4e93-8187-0ebf683dc069/volumes"
Oct 30 16:45:16.302522 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:16.302509    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"]
Oct 30 16:45:16.302882 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:16.302864    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-27r5k"]
Oct 30 16:45:17.260457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:17.260419    4497 generic.go:356] "Generic (PLEG): container finished" podID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerID="513c1ae34784ac4724b9b4c93ac3e38ed04c3034e8f68110dd36c1829a4992c7" exitCode=60
Oct 30 16:45:17.260924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:17.260528    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss" event={"ID":"7e4d51da-a8a2-4554-94fd-180a1704ba03","Type":"ContainerDied","Data":"513c1ae34784ac4724b9b4c93ac3e38ed04c3034e8f68110dd36c1829a4992c7"}
Oct 30 16:45:17.260924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:17.260630    4497 scope.go:122] "RemoveContainer" containerID="23d1a5ff2ce9ad3ce149ad2754d42bef2a72b71181a78472da69b63534ab1fb6"
Oct 30 16:45:17.261071 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:17.261051    4497 scope.go:122] "RemoveContainer" containerID="513c1ae34784ac4724b9b4c93ac3e38ed04c3034e8f68110dd36c1829a4992c7"
Oct 30 16:45:17.261430 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:17.261403    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss_odh-ai-gateway-infra(7e4d51da-a8a2-4554-94fd-180a1704ba03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03"
Oct 30 16:45:17.293905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:17.293872    4497 scope.go:122] "RemoveContainer" containerID="196ca0d6b7dc31b4670f22f62390747dc6164496efb9cfe3cc3934d0d7b465e7"
Oct 30 16:45:18.271524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.271491    4497 generic.go:356] "Generic (PLEG): container finished" podID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerID="272746eb5ae5508727819af5f4e4496bb4c786f8a2f5f39a0e79255be0516110" exitCode=60
Oct 30 16:45:18.271895 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.271567    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" event={"ID":"180063f7-7cf4-4c66-8e60-2b40551028c2","Type":"ContainerDied","Data":"272746eb5ae5508727819af5f4e4496bb4c786f8a2f5f39a0e79255be0516110"}
Oct 30 16:45:18.271895 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.271649    4497 scope.go:122] "RemoveContainer" containerID="196ca0d6b7dc31b4670f22f62390747dc6164496efb9cfe3cc3934d0d7b465e7"
Oct 30 16:45:18.272066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.272049    4497 scope.go:122] "RemoveContainer" containerID="272746eb5ae5508727819af5f4e4496bb4c786f8a2f5f39a0e79255be0516110"
Oct 30 16:45:18.272437 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:18.272414    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft_odh-ai-gateway-infra(180063f7-7cf4-4c66-8e60-2b40551028c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2"
Oct 30 16:45:18.300925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.300896    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82453041-c363-4a8a-be3d-42553041ebb9" path="/var/lib/kubelet/pods/82453041-c363-4a8a-be3d-42553041ebb9/volumes"
Oct 30 16:45:18.307872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.307830    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"]
Oct 30 16:45:18.480687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.480651    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:18.562018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.561983    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4d51da-a8a2-4554-94fd-180a1704ba03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4d51da-a8a2-4554-94fd-180a1704ba03-maas-api-service-ca\") pod \"7e4d51da-a8a2-4554-94fd-180a1704ba03\" (UID: \"7e4d51da-a8a2-4554-94fd-180a1704ba03\") "
Oct 30 16:45:18.562237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.562057    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4d51da-a8a2-4554-94fd-180a1704ba03-kube-api-access-6flpk\" (UniqueName: \"kubernetes.io/projected/7e4d51da-a8a2-4554-94fd-180a1704ba03-kube-api-access-6flpk\") pod \"7e4d51da-a8a2-4554-94fd-180a1704ba03\" (UID: \"7e4d51da-a8a2-4554-94fd-180a1704ba03\") "
Oct 30 16:45:18.562507 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.562477    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4d51da-a8a2-4554-94fd-180a1704ba03-maas-api-service-ca" pod "7e4d51da-a8a2-4554-94fd-180a1704ba03" (UID: "7e4d51da-a8a2-4554-94fd-180a1704ba03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:45:18.562817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.562795    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4d51da-a8a2-4554-94fd-180a1704ba03-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:18.564379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.564354    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4d51da-a8a2-4554-94fd-180a1704ba03-kube-api-access-6flpk" pod "7e4d51da-a8a2-4554-94fd-180a1704ba03" (UID: "7e4d51da-a8a2-4554-94fd-180a1704ba03"). InnerVolumeSpecName "kube-api-access-6flpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:45:18.664728 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:18.664694    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6flpk\" (UniqueName: \"kubernetes.io/projected/7e4d51da-a8a2-4554-94fd-180a1704ba03-kube-api-access-6flpk\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:19.281667 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.281629    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss" event={"ID":"7e4d51da-a8a2-4554-94fd-180a1704ba03","Type":"ContainerDied","Data":"3fe54a2cfc4ec301db514df4097cf62f2aef625dd4d584df23588899cd20f855"}
Oct 30 16:45:19.282263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.281698    4497 scope.go:122] "RemoveContainer" containerID="513c1ae34784ac4724b9b4c93ac3e38ed04c3034e8f68110dd36c1829a4992c7"
Oct 30 16:45:19.282263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.281788    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"
Oct 30 16:45:19.362622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.362066    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"]
Oct 30 16:45:19.388339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.388307    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"]
Oct 30 16:45:19.412297 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.412247    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4a4c68-29889645-7ktss"]
Oct 30 16:45:19.580579 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.580551    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:19.682259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.682220    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180063f7-7cf4-4c66-8e60-2b40551028c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180063f7-7cf4-4c66-8e60-2b40551028c2-maas-api-service-ca\") pod \"180063f7-7cf4-4c66-8e60-2b40551028c2\" (UID: \"180063f7-7cf4-4c66-8e60-2b40551028c2\") "
Oct 30 16:45:19.682464 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.682402    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180063f7-7cf4-4c66-8e60-2b40551028c2-kube-api-access-wm57h\" (UniqueName: \"kubernetes.io/projected/180063f7-7cf4-4c66-8e60-2b40551028c2-kube-api-access-wm57h\") pod \"180063f7-7cf4-4c66-8e60-2b40551028c2\" (UID: \"180063f7-7cf4-4c66-8e60-2b40551028c2\") "
Oct 30 16:45:19.682641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.682615    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180063f7-7cf4-4c66-8e60-2b40551028c2-maas-api-service-ca" pod "180063f7-7cf4-4c66-8e60-2b40551028c2" (UID: "180063f7-7cf4-4c66-8e60-2b40551028c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:45:19.682828 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.682813    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180063f7-7cf4-4c66-8e60-2b40551028c2-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:19.684500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.684474    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180063f7-7cf4-4c66-8e60-2b40551028c2-kube-api-access-wm57h" pod "180063f7-7cf4-4c66-8e60-2b40551028c2" (UID: "180063f7-7cf4-4c66-8e60-2b40551028c2"). InnerVolumeSpecName "kube-api-access-wm57h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:45:19.784331 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:19.784296    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm57h\" (UniqueName: \"kubernetes.io/projected/180063f7-7cf4-4c66-8e60-2b40551028c2-kube-api-access-wm57h\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:45:20.301371 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:20.300598    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"
Oct 30 16:45:20.304295 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:20.304250    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" path="/var/lib/kubelet/pods/7e4d51da-a8a2-4554-94fd-180a1704ba03/volumes"
Oct 30 16:45:20.305004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:20.304973    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft" event={"ID":"180063f7-7cf4-4c66-8e60-2b40551028c2","Type":"ContainerDied","Data":"3e07358fdc14e99b5cc715ba087957f4a54e6aa62573ce1c4496bf49f6862938"}
Oct 30 16:45:20.305076 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:20.305039    4497 scope.go:122] "RemoveContainer" containerID="272746eb5ae5508727819af5f4e4496bb4c786f8a2f5f39a0e79255be0516110"
Oct 30 16:45:20.353477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:20.353440    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"]
Oct 30 16:45:20.358022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:20.357985    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a98d1-29889645-bjrft"]
Oct 30 16:45:21.917627 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:21.917592    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:45:21.944060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944024    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"]
Oct 30 16:45:21.944718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944698    4497 cpu_manager.go:414] "containerMap: removing container" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944718    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944739    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5cb5996c-426c-48d9-8623-f6cd46b90b7f" containerName="payload-processing"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944747    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb5996c-426c-48d9-8623-f6cd46b90b7f" containerName="payload-processing"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944761    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944768    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944785    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944793    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944802    4497 cpu_manager.go:414] "containerMap: removing container" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.944817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944809    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944824    4497 cpu_manager.go:414] "containerMap: removing container" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944832    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944855    4497 cpu_manager.go:414] "containerMap: removing container" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944862    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944876    4497 cpu_manager.go:414] "containerMap: removing container" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944884    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944894    4497 cpu_manager.go:414] "containerMap: removing container" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944900    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944922    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e08ba398-0330-4e93-8187-0ebf683dc069" containerName="payload-pre-processing"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.944932    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08ba398-0330-4e93-8187-0ebf683dc069" containerName="payload-pre-processing"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945091    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945104    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945114    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="82453041-c363-4a8a-be3d-42553041ebb9" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945127    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08ba398-0330-4e93-8187-0ebf683dc069" containerName="payload-pre-processing"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945137    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945149    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945158    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb5996c-426c-48d9-8623-f6cd46b90b7f" containerName="payload-processing"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945167    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945178    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945203    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.945298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.945214    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" containerName="cleanup"
Oct 30 16:45:21.950204 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.950172    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:21.979602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.979570    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-dc1ea0-serving-cert\""
Oct 30 16:45:21.983958 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:21.983931    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"]
Oct 30 16:45:22.012279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.012239    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jg9x\" (UniqueName: \"kubernetes.io/projected/c7442bea-8f57-4357-8a14-068130d772c6-kube-api-access-9jg9x\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.012465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.012411    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.012622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.012602    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-metrics-certs\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.012820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.012799    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442bea-8f57-4357-8a14-068130d772c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.114283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.114243    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-metrics-certs\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.114481 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.114424    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442bea-8f57-4357-8a14-068130d772c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.114481 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.114453    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jg9x\" (UniqueName: \"kubernetes.io/projected/c7442bea-8f57-4357-8a14-068130d772c6-kube-api-access-9jg9x\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.114608 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.114535    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.114745 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:22.114713    4497 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-serving-cert: secret "maas-api-e2e-ipp-b-dc1ea0-serving-cert" not found
Oct 30 16:45:22.114862 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:22.114780    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls podName:c7442bea-8f57-4357-8a14-068130d772c6 nodeName:}" failed. No retries permitted until 2026-10-30 16:45:22.614763431 +0000 UTC m=+1352.468641675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls") pod "maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2" (UID: "c7442bea-8f57-4357-8a14-068130d772c6") : secret "maas-api-e2e-ipp-b-dc1ea0-serving-cert" not found
Oct 30 16:45:22.115021 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.115002    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442bea-8f57-4357-8a14-068130d772c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.116810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.116785    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-metrics-certs\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.135471 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.135438    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jg9x\" (UniqueName: \"kubernetes.io/projected/c7442bea-8f57-4357-8a14-068130d772c6-kube-api-access-9jg9x\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.301740 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.301697    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180063f7-7cf4-4c66-8e60-2b40551028c2" path="/var/lib/kubelet/pods/180063f7-7cf4-4c66-8e60-2b40551028c2/volumes"
Oct 30 16:45:22.622754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.622668    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.627304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.627251    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:22.864482 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:22.864444    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:23.171515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:23.171436    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"]
Oct 30 16:45:23.174930 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:45:23.174879    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7442bea_8f57_4357_8a14_068130d772c6.slice/crio-509cf427618693144e0846207d97bc4b59c385eb3a576851ef18127e94adf816 WatchSource:0}: Error finding container 509cf427618693144e0846207d97bc4b59c385eb3a576851ef18127e94adf816: Status 404 returned error can't find the container with id 509cf427618693144e0846207d97bc4b59c385eb3a576851ef18127e94adf816
Oct 30 16:45:23.176587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:23.176566    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:45:23.360025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:23.359988    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2" event={"ID":"c7442bea-8f57-4357-8a14-068130d772c6","Type":"ContainerStarted","Data":"509cf427618693144e0846207d97bc4b59c385eb3a576851ef18127e94adf816"}
Oct 30 16:45:24.368610 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:24.368566    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2" event={"ID":"c7442bea-8f57-4357-8a14-068130d772c6","Type":"ContainerStarted","Data":"dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9"}
Oct 30 16:45:24.368999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:24.368653    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:24.405766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:24.405714    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2" podStartSLOduration=2.981030905 podStartE2EDuration="3.405699478s" totalImagesPullingTime="424.668573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:24.405353027 +0000 UTC m=+1354.259231292" watchObservedRunningTime="2026-10-30 16:45:24.405699478 +0000 UTC m=+1354.259577742"
Oct 30 16:45:30.380375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:30.380343    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:45:34.601815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.601771    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"]
Oct 30 16:45:34.602485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.602468    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:34.602544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.602486    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4d51da-a8a2-4554-94fd-180a1704ba03" containerName="cleanup"
Oct 30 16:45:34.607312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.607294    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.621107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.621080    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f9970a-serving-cert\""
Oct 30 16:45:34.634225 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.634181    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"]
Oct 30 16:45:34.796379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.796343    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kps6\" (UniqueName: \"kubernetes.io/projected/dc80910c-0735-4a9e-8d6f-1f902c0738cc-kube-api-access-8kps6\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.796624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.796417    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.796624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.796570    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc80910c-0735-4a9e-8d6f-1f902c0738cc-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.796774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.796733    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-metrics-certs\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.898261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.898122    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kps6\" (UniqueName: \"kubernetes.io/projected/dc80910c-0735-4a9e-8d6f-1f902c0738cc-kube-api-access-8kps6\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.898261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.898240    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.898530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.898310    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc80910c-0735-4a9e-8d6f-1f902c0738cc-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.898530 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:34.898395    4497 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-serving-cert: secret "maas-api-e2e-api-b-f9970a-serving-cert" not found
Oct 30 16:45:34.898530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.898435    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-metrics-certs\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.898530 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:34.898469    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls podName:dc80910c-0735-4a9e-8d6f-1f902c0738cc nodeName:}" failed. No retries permitted until 2026-10-30 16:45:35.398446492 +0000 UTC m=+1365.252324743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls") pod "maas-api-e2e-api-b-f9970a-555985859d-29sj8" (UID: "dc80910c-0735-4a9e-8d6f-1f902c0738cc") : secret "maas-api-e2e-api-b-f9970a-serving-cert" not found
Oct 30 16:45:34.899119 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.899083    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc80910c-0735-4a9e-8d6f-1f902c0738cc-openshift-service-ca\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.901224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.901178    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-metrics-certs\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:34.921050 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:34.921024    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kps6\" (UniqueName: \"kubernetes.io/projected/dc80910c-0735-4a9e-8d6f-1f902c0738cc-kube-api-access-8kps6\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:35.406609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.406567    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:35.409173 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.409145    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls\") pod \"maas-api-e2e-api-b-f9970a-555985859d-29sj8\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:35.528792 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.528755    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:35.605237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.593085    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"]
Oct 30 16:45:35.625215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.621312    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.625215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.622662    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"]
Oct 30 16:45:35.632653 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.632620    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfr4k\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-kube-api-access-hfr4k\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.633513 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.633487    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f5c38f-62d7-4096-8f06-c90062d15a82-istiod-ca-cert\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.633648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.633585    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-certs\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.633648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.633613    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-data\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.633766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.633660    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-credential-socket\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.633766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.633685    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-envoy\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.633766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.633737    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-socket\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.634017 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.633874    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-token\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.634017 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.634000    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-podinfo\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.639692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.634463    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-06ac18-openshift-default-dockercfg-l84sf\""
Oct 30 16:45:35.735628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.735594    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-token\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.735819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.735651    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-podinfo\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.735819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.735771    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfr4k\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-kube-api-access-hfr4k\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.735819 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.735813    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f5c38f-62d7-4096-8f06-c90062d15a82-istiod-ca-cert\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.736364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.735884    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-certs\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.736364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.735922    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-data\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.736364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.735954    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-credential-socket\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.736364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.735986    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-envoy\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.736364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.736031    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-socket\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.736644 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.736533    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-socket\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.736841 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.736815    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-certs\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.737040 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.737016    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f5c38f-62d7-4096-8f06-c90062d15a82-istiod-ca-cert\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.737107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.737078    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-data\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.737383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.737356    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-credential-socket\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.740361 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.740336    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-podinfo\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.741474 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.741451    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-envoy\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.760226 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.760136    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-token\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.761050 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.761027    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfr4k\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-kube-api-access-hfr4k\") pod \"e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") " pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.982732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.982647    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:35.987439 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:35.987416    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"]
Oct 30 16:45:35.994567 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:45:35.994544    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc80910c_0735_4a9e_8d6f_1f902c0738cc.slice/crio-dd741346f4481cadf810f1fec27f0c2391e668a0ea7a3e6b5311c68084feb769 WatchSource:0}: Error finding container dd741346f4481cadf810f1fec27f0c2391e668a0ea7a3e6b5311c68084feb769: Status 404 returned error can't find the container with id dd741346f4481cadf810f1fec27f0c2391e668a0ea7a3e6b5311c68084feb769
Oct 30 16:45:36.310143 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.310108    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"]
Oct 30 16:45:36.313904 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:45:36.313869    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f5c38f_62d7_4096_8f06_c90062d15a82.slice/crio-0dec42eed6e705d03187fa57a8c0e2ae564b10274bb8d3901a9aa23ba67cb99f WatchSource:0}: Error finding container 0dec42eed6e705d03187fa57a8c0e2ae564b10274bb8d3901a9aa23ba67cb99f: Status 404 returned error can't find the container with id 0dec42eed6e705d03187fa57a8c0e2ae564b10274bb8d3901a9aa23ba67cb99f
Oct 30 16:45:36.316562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.316531    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:45:36.316631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.316595    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:45:36.316668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.316627    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:45:36.453784 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.453667    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" event={"ID":"f5f5c38f-62d7-4096-8f06-c90062d15a82","Type":"ContainerStarted","Data":"f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae"}
Oct 30 16:45:36.453987 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.453811    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" event={"ID":"f5f5c38f-62d7-4096-8f06-c90062d15a82","Type":"ContainerStarted","Data":"0dec42eed6e705d03187fa57a8c0e2ae564b10274bb8d3901a9aa23ba67cb99f"}
Oct 30 16:45:36.456015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.455986    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8" event={"ID":"dc80910c-0735-4a9e-8d6f-1f902c0738cc","Type":"ContainerStarted","Data":"699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5"}
Oct 30 16:45:36.456128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.456032    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8" event={"ID":"dc80910c-0735-4a9e-8d6f-1f902c0738cc","Type":"ContainerStarted","Data":"dd741346f4481cadf810f1fec27f0c2391e668a0ea7a3e6b5311c68084feb769"}
Oct 30 16:45:36.456128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.456085    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:36.498671 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.498617    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" podStartSLOduration=1.498597769 podStartE2EDuration="1.498597769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:36.495050566 +0000 UTC m=+1366.348928831" watchObservedRunningTime="2026-10-30 16:45:36.498597769 +0000 UTC m=+1366.352476034"
Oct 30 16:45:36.534576 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.534515    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8" podStartSLOduration=2.157304504 podStartE2EDuration="2.534497265s" totalImagesPullingTime="377.192761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:36.532436059 +0000 UTC m=+1366.386314324" watchObservedRunningTime="2026-10-30 16:45:36.534497265 +0000 UTC m=+1366.388375531"
Oct 30 16:45:36.983611 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:36.983580    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:37.989418 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:37.989386    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:38.469146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:38.469104    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:38.470215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:38.470171    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:45:40.963998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:40.963965    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"]
Oct 30 16:45:40.967939 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:40.967922    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:40.981865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:40.981843    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-06ac18\""
Oct 30 16:45:40.991361 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:40.991333    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-06ac18-dockercfg-zbnrz\""
Oct 30 16:45:40.996831 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:40.995544    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"]
Oct 30 16:45:41.023014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:41.022977    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8082cf07-73a5-4469-9436-59d3d2f9eba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz\" (UID: \"8082cf07-73a5-4469-9436-59d3d2f9eba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:41.023515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:41.023491    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9cmf\" (UniqueName: \"kubernetes.io/projected/8082cf07-73a5-4469-9436-59d3d2f9eba1-kube-api-access-t9cmf\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz\" (UID: \"8082cf07-73a5-4469-9436-59d3d2f9eba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:41.140210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:41.139407    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8082cf07-73a5-4469-9436-59d3d2f9eba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz\" (UID: \"8082cf07-73a5-4469-9436-59d3d2f9eba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:41.140210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:41.139622    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cmf\" (UniqueName: \"kubernetes.io/projected/8082cf07-73a5-4469-9436-59d3d2f9eba1-kube-api-access-t9cmf\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz\" (UID: \"8082cf07-73a5-4469-9436-59d3d2f9eba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:41.144208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:41.140938    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8082cf07-73a5-4469-9436-59d3d2f9eba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz\" (UID: \"8082cf07-73a5-4469-9436-59d3d2f9eba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:41.174048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:41.172210    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9cmf\" (UniqueName: \"kubernetes.io/projected/8082cf07-73a5-4469-9436-59d3d2f9eba1-kube-api-access-t9cmf\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz\" (UID: \"8082cf07-73a5-4469-9436-59d3d2f9eba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:41.281877 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:41.281843    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:41.592012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:41.591926    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"]
Oct 30 16:45:41.595146 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:45:41.595107    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8082cf07_73a5_4469_9436_59d3d2f9eba1.slice/crio-8d9e7fd70c5ea666c6e42110883c0157478338096438ce72529a9c63b844c7dd WatchSource:0}: Error finding container 8d9e7fd70c5ea666c6e42110883c0157478338096438ce72529a9c63b844c7dd: Status 404 returned error can't find the container with id 8d9e7fd70c5ea666c6e42110883c0157478338096438ce72529a9c63b844c7dd
Oct 30 16:45:42.467706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:42.467679    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:45:42.495564 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:42.495527    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz" event={"ID":"8082cf07-73a5-4469-9436-59d3d2f9eba1","Type":"ContainerStarted","Data":"50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697"}
Oct 30 16:45:42.495720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:42.495586    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz" event={"ID":"8082cf07-73a5-4469-9436-59d3d2f9eba1","Type":"ContainerStarted","Data":"8d9e7fd70c5ea666c6e42110883c0157478338096438ce72529a9c63b844c7dd"}
Oct 30 16:45:42.495720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:42.495613    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:45:42.550227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:42.546766    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz" podStartSLOduration=2.546704854 podStartE2EDuration="2.546704854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:42.544896088 +0000 UTC m=+1372.398774354" watchObservedRunningTime="2026-10-30 16:45:42.546704854 +0000 UTC m=+1372.400583119"
Oct 30 16:45:51.916955 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:45:51.916918    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:45:53.505037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:45:53.505009    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:46:21.916691 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:46:21.916650    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:46:27.172932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:27.172893    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"]
Oct 30 16:46:27.209088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:27.209052    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"]
Oct 30 16:46:30.859527 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:30.859490    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"]
Oct 30 16:46:30.860162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:30.860129    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr" podUID="4cbc1604-9a77-4f6e-adea-2605b83d21c5" containerName="maas-api" containerID="cri-o://ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b" gracePeriod=30
Oct 30 16:46:31.258450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.258421    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:46:31.347820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.347784    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbc1604-9a77-4f6e-adea-2605b83d21c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbc1604-9a77-4f6e-adea-2605b83d21c5-openshift-service-ca\") pod \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") "
Oct 30 16:46:31.348024 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.348007    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-metrics-certs\") pod \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") "
Oct 30 16:46:31.348199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.348154    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls\") pod \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") "
Oct 30 16:46:31.348342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.348203    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbc1604-9a77-4f6e-adea-2605b83d21c5-kube-api-access-qcczm\" (UniqueName: \"kubernetes.io/projected/4cbc1604-9a77-4f6e-adea-2605b83d21c5-kube-api-access-qcczm\") pod \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\" (UID: \"4cbc1604-9a77-4f6e-adea-2605b83d21c5\") "
Oct 30 16:46:31.348342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.348237    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbc1604-9a77-4f6e-adea-2605b83d21c5-openshift-service-ca" pod "4cbc1604-9a77-4f6e-adea-2605b83d21c5" (UID: "4cbc1604-9a77-4f6e-adea-2605b83d21c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:46:31.348791 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.348765    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbc1604-9a77-4f6e-adea-2605b83d21c5-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:31.350929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.350887    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls" pod "4cbc1604-9a77-4f6e-adea-2605b83d21c5" (UID: "4cbc1604-9a77-4f6e-adea-2605b83d21c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:46:31.352059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.351471    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbc1604-9a77-4f6e-adea-2605b83d21c5-kube-api-access-qcczm" pod "4cbc1604-9a77-4f6e-adea-2605b83d21c5" (UID: "4cbc1604-9a77-4f6e-adea-2605b83d21c5"). InnerVolumeSpecName "kube-api-access-qcczm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:46:31.353425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.353393    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-metrics-certs" pod "4cbc1604-9a77-4f6e-adea-2605b83d21c5" (UID: "4cbc1604-9a77-4f6e-adea-2605b83d21c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:46:31.452617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.452561    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:31.452617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.452617    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbc1604-9a77-4f6e-adea-2605b83d21c5-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:31.452884 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.452633    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcczm\" (UniqueName: \"kubernetes.io/projected/4cbc1604-9a77-4f6e-adea-2605b83d21c5-kube-api-access-qcczm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:31.928076 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.928044    4497 generic.go:356] "Generic (PLEG): container finished" podID="4cbc1604-9a77-4f6e-adea-2605b83d21c5" containerID="ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b" exitCode=0
Oct 30 16:46:31.928572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.928096    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"
Oct 30 16:46:31.928572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.928128    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr" event={"ID":"4cbc1604-9a77-4f6e-adea-2605b83d21c5","Type":"ContainerDied","Data":"ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b"}
Oct 30 16:46:31.928572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.928203    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr" event={"ID":"4cbc1604-9a77-4f6e-adea-2605b83d21c5","Type":"ContainerDied","Data":"54631a9537aee642bbbbf170ac33896d8b4b60ea093795030e91fbae48514f6d"}
Oct 30 16:46:31.928572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.928238    4497 scope.go:122] "RemoveContainer" containerID="ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b"
Oct 30 16:46:31.948864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.948840    4497 scope.go:122] "RemoveContainer" containerID="ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b"
Oct 30 16:46:31.949332 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:46:31.949289    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b\": container with ID starting with ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b not found: ID does not exist" containerID="ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b"
Oct 30 16:46:31.949452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:31.949330    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b"} err="failed to get container status \"ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b\": rpc error: code = NotFound desc = could not find container \"ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b\": container with ID starting with ba8df516a52b96ba00a83c5f229062307366108db6f5bf68e17fe5a01e0ada3b not found: ID does not exist"
Oct 30 16:46:32.005221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.005164    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"]
Oct 30 16:46:32.008249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.008220    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0a98d1-7598b78567-fx2kr"]
Oct 30 16:46:32.176897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.176251    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j" podUID="f8aba258-c5a6-4e67-9c6f-0037d6a61193" containerName="payload-processing" containerID="cri-o://089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e" gracePeriod=25
Oct 30 16:46:32.210143 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.210090    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67" podUID="9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3" containerName="payload-pre-processing" containerID="cri-o://adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2" gracePeriod=25
Oct 30 16:46:32.304766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.304733    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbc1604-9a77-4f6e-adea-2605b83d21c5" path="/var/lib/kubelet/pods/4cbc1604-9a77-4f6e-adea-2605b83d21c5/volumes"
Oct 30 16:46:32.556019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.555989    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:46:32.624390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.624360    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:46:32.669686 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.669651    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8aba258-c5a6-4e67-9c6f-0037d6a61193-kube-api-access-xjjlw\" (UniqueName: \"kubernetes.io/projected/f8aba258-c5a6-4e67-9c6f-0037d6a61193-kube-api-access-xjjlw\") pod \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\" (UID: \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\") "
Oct 30 16:46:32.669990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.669973    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8aba258-c5a6-4e67-9c6f-0037d6a61193-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8aba258-c5a6-4e67-9c6f-0037d6a61193-plugins-config-volume\") pod \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\" (UID: \"f8aba258-c5a6-4e67-9c6f-0037d6a61193\") "
Oct 30 16:46:32.670444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.670414    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8aba258-c5a6-4e67-9c6f-0037d6a61193-plugins-config-volume" pod "f8aba258-c5a6-4e67-9c6f-0037d6a61193" (UID: "f8aba258-c5a6-4e67-9c6f-0037d6a61193"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:46:32.670606 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.670589    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8aba258-c5a6-4e67-9c6f-0037d6a61193-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:32.701067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.701028    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8aba258-c5a6-4e67-9c6f-0037d6a61193-kube-api-access-xjjlw" pod "f8aba258-c5a6-4e67-9c6f-0037d6a61193" (UID: "f8aba258-c5a6-4e67-9c6f-0037d6a61193"). InnerVolumeSpecName "kube-api-access-xjjlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:46:32.788209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.785639    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-kube-api-access-d5kpq\" (UniqueName: \"kubernetes.io/projected/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-kube-api-access-d5kpq\") pod \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\" (UID: \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\") "
Oct 30 16:46:32.788209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.786009    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-plugins-config-volume\") pod \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\" (UID: \"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3\") "
Oct 30 16:46:32.788209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.786598    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjjlw\" (UniqueName: \"kubernetes.io/projected/f8aba258-c5a6-4e67-9c6f-0037d6a61193-kube-api-access-xjjlw\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:32.796495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.796433    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-plugins-config-volume" pod "9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3" (UID: "9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:46:32.797156 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.797129    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-kube-api-access-d5kpq" pod "9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3" (UID: "9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3"). InnerVolumeSpecName "kube-api-access-d5kpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:46:32.894212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.893390    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:32.894212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.893445    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kpq\" (UniqueName: \"kubernetes.io/projected/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3-kube-api-access-d5kpq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:32.936214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.936165    4497 generic.go:356] "Generic (PLEG): container finished" podID="f8aba258-c5a6-4e67-9c6f-0037d6a61193" containerID="089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e" exitCode=0
Oct 30 16:46:32.936688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.936225    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"
Oct 30 16:46:32.936688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.936252    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j" event={"ID":"f8aba258-c5a6-4e67-9c6f-0037d6a61193","Type":"ContainerDied","Data":"089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e"}
Oct 30 16:46:32.936688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.936319    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j" event={"ID":"f8aba258-c5a6-4e67-9c6f-0037d6a61193","Type":"ContainerDied","Data":"068f1a4f9c004f376f9a8dcfcf7f3c9cd508aaeb43856d97234bcc6983601f8c"}
Oct 30 16:46:32.936688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.936352    4497 scope.go:122] "RemoveContainer" containerID="089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e"
Oct 30 16:46:32.938936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.938912    4497 generic.go:356] "Generic (PLEG): container finished" podID="9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3" containerID="adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2" exitCode=0
Oct 30 16:46:32.939076 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.938947    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67" event={"ID":"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3","Type":"ContainerDied","Data":"adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2"}
Oct 30 16:46:32.939076 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.938969    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"
Oct 30 16:46:32.939076 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.939002    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67" event={"ID":"9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3","Type":"ContainerDied","Data":"ee4a234dc65533f790d7413b85acf6609ace0456cca01367f4eab90cca60212f"}
Oct 30 16:46:32.948099 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.948073    4497 scope.go:122] "RemoveContainer" containerID="089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e"
Oct 30 16:46:32.948447 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:46:32.948424    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e\": container with ID starting with 089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e not found: ID does not exist" containerID="089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e"
Oct 30 16:46:32.948537 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.948455    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e"} err="failed to get container status \"089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e\": rpc error: code = NotFound desc = could not find container \"089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e\": container with ID starting with 089ce3ca0e6a3cb9654c7525bb371fcf5f86b495e601923acf175ea0d712281e not found: ID does not exist"
Oct 30 16:46:32.948537 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.948474    4497 scope.go:122] "RemoveContainer" containerID="adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2"
Oct 30 16:46:32.958176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.958155    4497 scope.go:122] "RemoveContainer" containerID="adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2"
Oct 30 16:46:32.958537 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:46:32.958515    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2\": container with ID starting with adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2 not found: ID does not exist" containerID="adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2"
Oct 30 16:46:32.958589 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:32.958553    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2"} err="failed to get container status \"adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2\": rpc error: code = NotFound desc = could not find container \"adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2\": container with ID starting with adb30e6b8939a7bd9ac5de61098beda38a9cf96384f8fb18c50851df189862c2 not found: ID does not exist"
Oct 30 16:46:33.001360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:33.001319    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"]
Oct 30 16:46:33.010316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:33.010287    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08fcc8-5b5c7b7b8b-6tz7j"]
Oct 30 16:46:33.053157 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:33.053078    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"]
Oct 30 16:46:33.056616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:33.056591    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08fcc8-546567fb7f-68k67"]
Oct 30 16:46:34.301683 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:34.301653    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3" path="/var/lib/kubelet/pods/9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3/volumes"
Oct 30 16:46:34.302124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:34.302110    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8aba258-c5a6-4e67-9c6f-0037d6a61193" path="/var/lib/kubelet/pods/f8aba258-c5a6-4e67-9c6f-0037d6a61193/volumes"
Oct 30 16:46:38.009619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:38.009582    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerID="f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae" exitCode=0
Oct 30 16:46:38.010305 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:38.010279    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" event={"ID":"f5f5c38f-62d7-4096-8f06-c90062d15a82","Type":"ContainerDied","Data":"f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae"}
Oct 30 16:46:38.011012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:38.010973    4497 scope.go:122] "RemoveContainer" containerID="f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae"
Oct 30 16:46:38.014024 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:38.013985    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:46:38.014226 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:38.014207    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:46:38.014515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:38.014495    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:46:38.469568 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:38.469479    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:46:39.018550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:39.018517    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" event={"ID":"f5f5c38f-62d7-4096-8f06-c90062d15a82","Type":"ContainerStarted","Data":"c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf"}
Oct 30 16:46:39.994209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:39.991557    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:46:40.009253 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:40.000387    4497 patch_prober.go:29] interesting pod/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 16:46:40.009253 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:40.000479    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 16:46:40.155645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:40.155068    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"]
Oct 30 16:46:40.155645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:40.155373    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy" containerID="cri-o://b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3" gracePeriod=30
Oct 30 16:46:40.996215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:40.985906    4497 patch_prober.go:29] interesting pod/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 16:46:40.996215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:40.985968    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 16:46:41.991210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:41.986593    4497 patch_prober.go:29] interesting pod/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 16:46:41.991210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:41.986660    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 16:46:42.983870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:42.983393    4497 patch_prober.go:29] interesting pod/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 16:46:42.983870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:42.983459    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 16:46:43.258064 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:43.258030    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:46:43.258485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:43.258080    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:46:43.996641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:43.996604    4497 patch_prober.go:29] interesting pod/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Oct 30 16:46:43.996803 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:43.996670    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Oct 30 16:46:44.987884 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:44.987850    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:46:44.988315 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:44.988293    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:46:44.989006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:44.988984    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:46:45.500719 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.500695    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:46:45.590724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.590651    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-data\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.590724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.590682    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-certs\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.590973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.590741    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istiod-ca-cert\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.590973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.590782    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-credential-socket\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.590973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.590874    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-socket\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.590973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.590916    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-envoy\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.590973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.590952    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-token\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-token\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.591214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.590994    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-data" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:46:45.591214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591059    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-certs" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:46:45.591214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591085    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-kube-api-access-qwdsj\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-kube-api-access-qwdsj\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.591214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591126    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-podinfo\") pod \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\" (UID: \"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce\") "
Oct 30 16:46:45.591444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591212    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-socket" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:46:45.591444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591348    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-credential-socket" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:46:45.591621 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591589    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istiod-ca-cert" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:46:45.591916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591892    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:45.592004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591920    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:45.592004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591933    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:45.592004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591948    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:45.592004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.591960    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:45.593438 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.593415    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-token" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:46:45.593892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.593871    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-envoy" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:46:45.593986 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.593943    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-podinfo" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:46:45.594365 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.594342    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-kube-api-access-qwdsj" pod "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" (UID: "ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce"). InnerVolumeSpecName "kube-api-access-qwdsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:46:45.693762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.693732    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:45.693762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.693759    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:45.693762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.693769    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdsj\" (UniqueName: \"kubernetes.io/projected/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-kube-api-access-qwdsj\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:45.694005 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:45.693778    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:46.067128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.067088    4497 generic.go:356] "Generic (PLEG): container finished" podID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerID="b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3" exitCode=0
Oct 30 16:46:46.067599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.067138    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"
Oct 30 16:46:46.067599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.067167    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" event={"ID":"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce","Type":"ContainerDied","Data":"b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3"}
Oct 30 16:46:46.067599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.067238    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2" event={"ID":"ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce","Type":"ContainerDied","Data":"92824e864dbfc8112c20cfa08ae4ce39ce95eca46807a491771dab587a81bd53"}
Oct 30 16:46:46.067599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.067265    4497 scope.go:122] "RemoveContainer" containerID="b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3"
Oct 30 16:46:46.078522 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.078501    4497 scope.go:122] "RemoveContainer" containerID="600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb"
Oct 30 16:46:46.087458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.087436    4497 scope.go:122] "RemoveContainer" containerID="b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3"
Oct 30 16:46:46.087784 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:46:46.087762    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3\": container with ID starting with b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3 not found: ID does not exist" containerID="b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3"
Oct 30 16:46:46.087847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.087797    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3"} err="failed to get container status \"b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3\": rpc error: code = NotFound desc = could not find container \"b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3\": container with ID starting with b73959e4024fd4ee97b945d9331973178bcdb88c2ffb42e11328c2f2c912dea3 not found: ID does not exist"
Oct 30 16:46:46.087847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.087815    4497 scope.go:122] "RemoveContainer" containerID="600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb"
Oct 30 16:46:46.088101 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:46:46.088081    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb\": container with ID starting with 600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb not found: ID does not exist" containerID="600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb"
Oct 30 16:46:46.088144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.088108    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb"} err="failed to get container status \"600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb\": rpc error: code = NotFound desc = could not find container \"600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb\": container with ID starting with 600dfbfc4fe2b0e2bee07f83f9e6e3d9e1211a4b285aafd47e54713a781382bb not found: ID does not exist"
Oct 30 16:46:46.112124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.112087    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"]
Oct 30 16:46:46.121238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.121209    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a98d1-openshift-default-5dfdcd49cc-58qx2"]
Oct 30 16:46:46.299865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:46.299835    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" path="/var/lib/kubelet/pods/ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce/volumes"
Oct 30 16:46:50.483704 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:50.483665    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"]
Oct 30 16:46:50.863199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:50.863155    4497 scope.go:122] "RemoveContainer" containerID="1cf8ff60e573d9a8957680e3c644975ce2903138e95530fa7919d21ff6052e25"
Oct 30 16:46:51.916684 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:46:51.916644    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:46:55.484929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:55.484882    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz" podUID="8082cf07-73a5-4469-9436-59d3d2f9eba1" containerName="payload-pre-processing" containerID="cri-o://50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697" gracePeriod=25
Oct 30 16:46:55.763678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:55.763652    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:46:55.942207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:55.942146    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8082cf07-73a5-4469-9436-59d3d2f9eba1-kube-api-access-t9cmf\" (UniqueName: \"kubernetes.io/projected/8082cf07-73a5-4469-9436-59d3d2f9eba1-kube-api-access-t9cmf\") pod \"8082cf07-73a5-4469-9436-59d3d2f9eba1\" (UID: \"8082cf07-73a5-4469-9436-59d3d2f9eba1\") "
Oct 30 16:46:55.942407 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:55.942255    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8082cf07-73a5-4469-9436-59d3d2f9eba1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8082cf07-73a5-4469-9436-59d3d2f9eba1-plugins-config-volume\") pod \"8082cf07-73a5-4469-9436-59d3d2f9eba1\" (UID: \"8082cf07-73a5-4469-9436-59d3d2f9eba1\") "
Oct 30 16:46:55.942678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:55.942653    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8082cf07-73a5-4469-9436-59d3d2f9eba1-plugins-config-volume" pod "8082cf07-73a5-4469-9436-59d3d2f9eba1" (UID: "8082cf07-73a5-4469-9436-59d3d2f9eba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:46:55.944371 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:55.944353    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8082cf07-73a5-4469-9436-59d3d2f9eba1-kube-api-access-t9cmf" pod "8082cf07-73a5-4469-9436-59d3d2f9eba1" (UID: "8082cf07-73a5-4469-9436-59d3d2f9eba1"). InnerVolumeSpecName "kube-api-access-t9cmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:46:56.052074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.046098    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9cmf\" (UniqueName: \"kubernetes.io/projected/8082cf07-73a5-4469-9436-59d3d2f9eba1-kube-api-access-t9cmf\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:56.052074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.046137    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8082cf07-73a5-4469-9436-59d3d2f9eba1-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:46:56.143817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.143782    4497 generic.go:356] "Generic (PLEG): container finished" podID="8082cf07-73a5-4469-9436-59d3d2f9eba1" containerID="50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697" exitCode=0
Oct 30 16:46:56.143981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.143835    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"
Oct 30 16:46:56.143981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.143861    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz" event={"ID":"8082cf07-73a5-4469-9436-59d3d2f9eba1","Type":"ContainerDied","Data":"50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697"}
Oct 30 16:46:56.143981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.143957    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz" event={"ID":"8082cf07-73a5-4469-9436-59d3d2f9eba1","Type":"ContainerDied","Data":"8d9e7fd70c5ea666c6e42110883c0157478338096438ce72529a9c63b844c7dd"}
Oct 30 16:46:56.144103 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.143991    4497 scope.go:122] "RemoveContainer" containerID="50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697"
Oct 30 16:46:56.156316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.156289    4497 scope.go:122] "RemoveContainer" containerID="50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697"
Oct 30 16:46:56.156667 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:46:56.156642    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697\": container with ID starting with 50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697 not found: ID does not exist" containerID="50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697"
Oct 30 16:46:56.156733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.156681    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697"} err="failed to get container status \"50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697\": rpc error: code = NotFound desc = could not find container \"50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697\": container with ID starting with 50da6d1dcf27c87ed6a3d189a7ce3cf230070439f636a69946fd4c0cf7fef697 not found: ID does not exist"
Oct 30 16:46:56.188619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.188586    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"]
Oct 30 16:46:56.198237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.198210    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-76867c8799-lsbpz"]
Oct 30 16:46:56.300830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:46:56.300800    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8082cf07-73a5-4469-9436-59d3d2f9eba1" path="/var/lib/kubelet/pods/8082cf07-73a5-4469-9436-59d3d2f9eba1/volumes"
Oct 30 16:47:07.990928 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:07.990889    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"]
Oct 30 16:47:07.991451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:07.991172    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy" containerID="cri-o://88367817e90f7f6ba6e7c443d1747a3819ae4a297c42ace13025269b2cbf3ade" gracePeriod=30
Oct 30 16:47:11.671814 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:11.671772    4497 patch_prober.go:29] interesting pod/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:47:11.672240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:11.671843    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:47:13.257282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.257252    4497 generic.go:356] "Generic (PLEG): container finished" podID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerID="88367817e90f7f6ba6e7c443d1747a3819ae4a297c42ace13025269b2cbf3ade" exitCode=0
Oct 30 16:47:13.257730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.257331    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" event={"ID":"f4f1d7ae-9027-4ee7-8972-bc953e335c04","Type":"ContainerDied","Data":"88367817e90f7f6ba6e7c443d1747a3819ae4a297c42ace13025269b2cbf3ade"}
Oct 30 16:47:13.257730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.257408    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9" event={"ID":"f4f1d7ae-9027-4ee7-8972-bc953e335c04","Type":"ContainerDied","Data":"5376636552b21bb259d2c5d6e94731ff6de372c7767ccf52f69ea742c864b32e"}
Oct 30 16:47:13.257730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.257438    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5376636552b21bb259d2c5d6e94731ff6de372c7767ccf52f69ea742c864b32e"
Oct 30 16:47:13.257730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.257461    4497 scope.go:122] "RemoveContainer" containerID="b0e69a0db9afaaaff31d18bb4c1d07140cf7cc1cfdda866063e091568ee0ce1e"
Oct 30 16:47:13.272001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.271979    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:47:13.421643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.421602    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-certs\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.421863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.421691    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-data\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.421863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.421732    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-socket\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.421863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.421813    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-kube-api-access-5mqsz\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-kube-api-access-5mqsz\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.421863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.421847    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-token\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.422130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.421872    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-envoy\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.422130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.421906    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-credential-socket\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.422130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.421910    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-certs" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:47:13.422130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422034    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-data" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:47:13.422130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422092    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-socket" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:47:13.422421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422141    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-podinfo\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.422421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422222    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istiod-ca-cert\") pod \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\" (UID: \"f4f1d7ae-9027-4ee7-8972-bc953e335c04\") "
Oct 30 16:47:13.422421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422281    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-credential-socket" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:47:13.422640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422609    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istiod-ca-cert" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:47:13.422888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422852    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:13.422982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422894    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:13.422982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422907    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:13.422982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422919    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:13.422982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.422930    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:13.424400 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.424377    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-envoy" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:47:13.424497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.424463    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-podinfo" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:47:13.424548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.424490    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-kube-api-access-5mqsz" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "kube-api-access-5mqsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:47:13.425398 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.425383    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-token" pod "f4f1d7ae-9027-4ee7-8972-bc953e335c04" (UID: "f4f1d7ae-9027-4ee7-8972-bc953e335c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:47:13.524538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.524460    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:13.524538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.524490    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mqsz\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-kube-api-access-5mqsz\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:13.524538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.524499    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:13.524538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:13.524508    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f1d7ae-9027-4ee7-8972-bc953e335c04-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:14.266560 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:14.266534    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"
Oct 30 16:47:14.352611 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:14.352574    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"]
Oct 30 16:47:14.360921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:14.360891    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-08fcc8-openshift-default-6cc6f8b6d9-zm6q9"]
Oct 30 16:47:16.300413 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:16.300376    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" path="/var/lib/kubelet/pods/f4f1d7ae-9027-4ee7-8972-bc953e335c04/volumes"
Oct 30 16:47:21.917013 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:47:21.916974    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:47:26.703488 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:26.703454    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"]
Oct 30 16:47:26.704062 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:26.703757    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2" podUID="c7442bea-8f57-4357-8a14-068130d772c6" containerName="maas-api" containerID="cri-o://dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9" gracePeriod=30
Oct 30 16:47:26.992693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:26.992665    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:47:27.049563 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.049525    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7442bea-8f57-4357-8a14-068130d772c6-kube-api-access-9jg9x\" (UniqueName: \"kubernetes.io/projected/c7442bea-8f57-4357-8a14-068130d772c6-kube-api-access-9jg9x\") pod \"c7442bea-8f57-4357-8a14-068130d772c6\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") "
Oct 30 16:47:27.049788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.049623    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls\") pod \"c7442bea-8f57-4357-8a14-068130d772c6\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") "
Oct 30 16:47:27.049788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.049693    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7442bea-8f57-4357-8a14-068130d772c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442bea-8f57-4357-8a14-068130d772c6-openshift-service-ca\") pod \"c7442bea-8f57-4357-8a14-068130d772c6\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") "
Oct 30 16:47:27.049922 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.049832    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-metrics-certs\") pod \"c7442bea-8f57-4357-8a14-068130d772c6\" (UID: \"c7442bea-8f57-4357-8a14-068130d772c6\") "
Oct 30 16:47:27.050670 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.050624    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7442bea-8f57-4357-8a14-068130d772c6-openshift-service-ca" pod "c7442bea-8f57-4357-8a14-068130d772c6" (UID: "c7442bea-8f57-4357-8a14-068130d772c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:47:27.052869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.052842    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-metrics-certs" pod "c7442bea-8f57-4357-8a14-068130d772c6" (UID: "c7442bea-8f57-4357-8a14-068130d772c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:47:27.053027 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.053009    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls" pod "c7442bea-8f57-4357-8a14-068130d772c6" (UID: "c7442bea-8f57-4357-8a14-068130d772c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:47:27.053132 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.053117    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7442bea-8f57-4357-8a14-068130d772c6-kube-api-access-9jg9x" pod "c7442bea-8f57-4357-8a14-068130d772c6" (UID: "c7442bea-8f57-4357-8a14-068130d772c6"). InnerVolumeSpecName "kube-api-access-9jg9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:47:27.152140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.152104    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:27.152140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.152134    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jg9x\" (UniqueName: \"kubernetes.io/projected/c7442bea-8f57-4357-8a14-068130d772c6-kube-api-access-9jg9x\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:27.152140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.152145    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7442bea-8f57-4357-8a14-068130d772c6-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:27.152417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.152153    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442bea-8f57-4357-8a14-068130d772c6-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:47:27.354499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.354462    4497 generic.go:356] "Generic (PLEG): container finished" podID="c7442bea-8f57-4357-8a14-068130d772c6" containerID="dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9" exitCode=0
Oct 30 16:47:27.354735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.354601    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2" event={"ID":"c7442bea-8f57-4357-8a14-068130d772c6","Type":"ContainerDied","Data":"dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9"}
Oct 30 16:47:27.354735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.354651    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2" event={"ID":"c7442bea-8f57-4357-8a14-068130d772c6","Type":"ContainerDied","Data":"509cf427618693144e0846207d97bc4b59c385eb3a576851ef18127e94adf816"}
Oct 30 16:47:27.354735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.354678    4497 scope.go:122] "RemoveContainer" containerID="dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9"
Oct 30 16:47:27.354927 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.354820    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"
Oct 30 16:47:27.371778 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.371751    4497 scope.go:122] "RemoveContainer" containerID="dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9"
Oct 30 16:47:27.372134 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:47:27.372109    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9\": container with ID starting with dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9 not found: ID does not exist" containerID="dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9"
Oct 30 16:47:27.372295 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.372140    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9"} err="failed to get container status \"dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9\": rpc error: code = NotFound desc = could not find container \"dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9\": container with ID starting with dc08fd29bb61feeaefa26a2d79dd7b081ca18dab6703d4c60fc8b598071fbaa9 not found: ID does not exist"
Oct 30 16:47:27.400614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.400576    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"]
Oct 30 16:47:27.406542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:27.406508    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dc1ea0-6485468887-h4mj2"]
Oct 30 16:47:28.300094 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:28.300061    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7442bea-8f57-4357-8a14-068130d772c6" path="/var/lib/kubelet/pods/c7442bea-8f57-4357-8a14-068130d772c6/volumes"
Oct 30 16:47:51.932844 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:47:51.932771    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:47:59.642823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:59.642787    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"]
Oct 30 16:47:59.643372 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:59.643011    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8" podUID="dc80910c-0735-4a9e-8d6f-1f902c0738cc" containerName="maas-api" containerID="cri-o://699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5" gracePeriod=30
Oct 30 16:47:59.970057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:47:59.970028    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:48:00.031158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.031124    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls\") pod \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") "
Oct 30 16:48:00.031158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.031165    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc80910c-0735-4a9e-8d6f-1f902c0738cc-kube-api-access-8kps6\" (UniqueName: \"kubernetes.io/projected/dc80910c-0735-4a9e-8d6f-1f902c0738cc-kube-api-access-8kps6\") pod \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") "
Oct 30 16:48:00.031490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.031340    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc80910c-0735-4a9e-8d6f-1f902c0738cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc80910c-0735-4a9e-8d6f-1f902c0738cc-openshift-service-ca\") pod \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") "
Oct 30 16:48:00.031490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.031382    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-metrics-certs\") pod \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\" (UID: \"dc80910c-0735-4a9e-8d6f-1f902c0738cc\") "
Oct 30 16:48:00.031995 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.031693    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc80910c-0735-4a9e-8d6f-1f902c0738cc-openshift-service-ca" pod "dc80910c-0735-4a9e-8d6f-1f902c0738cc" (UID: "dc80910c-0735-4a9e-8d6f-1f902c0738cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:00.031995 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.031954    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc80910c-0735-4a9e-8d6f-1f902c0738cc-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:00.034924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.034898    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-metrics-certs" pod "dc80910c-0735-4a9e-8d6f-1f902c0738cc" (UID: "dc80910c-0735-4a9e-8d6f-1f902c0738cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:00.035044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.034938    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls" pod "dc80910c-0735-4a9e-8d6f-1f902c0738cc" (UID: "dc80910c-0735-4a9e-8d6f-1f902c0738cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:48:00.035269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.035248    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc80910c-0735-4a9e-8d6f-1f902c0738cc-kube-api-access-8kps6" pod "dc80910c-0735-4a9e-8d6f-1f902c0738cc" (UID: "dc80910c-0735-4a9e-8d6f-1f902c0738cc"). InnerVolumeSpecName "kube-api-access-8kps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:00.133694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.133617    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:00.133694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.133651    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kps6\" (UniqueName: \"kubernetes.io/projected/dc80910c-0735-4a9e-8d6f-1f902c0738cc-kube-api-access-8kps6\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:00.133694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.133665    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc80910c-0735-4a9e-8d6f-1f902c0738cc-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:00.585988 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.585902    4497 generic.go:356] "Generic (PLEG): container finished" podID="dc80910c-0735-4a9e-8d6f-1f902c0738cc" containerID="699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5" exitCode=0
Oct 30 16:48:00.585988 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.585953    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"
Oct 30 16:48:00.586176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.585986    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8" event={"ID":"dc80910c-0735-4a9e-8d6f-1f902c0738cc","Type":"ContainerDied","Data":"699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5"}
Oct 30 16:48:00.586176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.586049    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8" event={"ID":"dc80910c-0735-4a9e-8d6f-1f902c0738cc","Type":"ContainerDied","Data":"dd741346f4481cadf810f1fec27f0c2391e668a0ea7a3e6b5311c68084feb769"}
Oct 30 16:48:00.586176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.586081    4497 scope.go:122] "RemoveContainer" containerID="699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5"
Oct 30 16:48:00.596104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.596082    4497 scope.go:122] "RemoveContainer" containerID="699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5"
Oct 30 16:48:00.596450 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:00.596423    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5\": container with ID starting with 699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5 not found: ID does not exist" containerID="699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5"
Oct 30 16:48:00.596561 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.596457    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5"} err="failed to get container status \"699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5\": rpc error: code = NotFound desc = could not find container \"699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5\": container with ID starting with 699e8bf63c3a7074dd463b4b4e71c5ba5ba277092655b897044aba1b743acbf5 not found: ID does not exist"
Oct 30 16:48:00.639796 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.639758    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"]
Oct 30 16:48:00.645708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:00.645673    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f9970a-555985859d-29sj8"]
Oct 30 16:48:02.324457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:02.324419    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc80910c-0735-4a9e-8d6f-1f902c0738cc" path="/var/lib/kubelet/pods/dc80910c-0735-4a9e-8d6f-1f902c0738cc/volumes"
Oct 30 16:48:10.039153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.039115    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"]
Oct 30 16:48:10.040019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.039997    4497 cpu_manager.go:414] "containerMap: removing container" podUID="8082cf07-73a5-4469-9436-59d3d2f9eba1" containerName="payload-pre-processing"
Oct 30 16:48:10.040078 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040022    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8082cf07-73a5-4469-9436-59d3d2f9eba1" containerName="payload-pre-processing"
Oct 30 16:48:10.040078 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040039    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy"
Oct 30 16:48:10.040078 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040047    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy"
Oct 30 16:48:10.040078 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040076    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040083    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040099    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040108    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040131    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c7442bea-8f57-4357-8a14-068130d772c6" containerName="maas-api"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040138    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7442bea-8f57-4357-8a14-068130d772c6" containerName="maas-api"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040155    4497 cpu_manager.go:414] "containerMap: removing container" podUID="dc80910c-0735-4a9e-8d6f-1f902c0738cc" containerName="maas-api"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040162    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc80910c-0735-4a9e-8d6f-1f902c0738cc" containerName="maas-api"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040179    4497 cpu_manager.go:414] "containerMap: removing container" podUID="4cbc1604-9a77-4f6e-adea-2605b83d21c5" containerName="maas-api"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040201    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbc1604-9a77-4f6e-adea-2605b83d21c5" containerName="maas-api"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040215    4497 cpu_manager.go:414] "containerMap: removing container" podUID="9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3" containerName="payload-pre-processing"
Oct 30 16:48:10.040237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040223    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3" containerName="payload-pre-processing"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040244    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f8aba258-c5a6-4e67-9c6f-0037d6a61193" containerName="payload-processing"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040251    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8aba258-c5a6-4e67-9c6f-0037d6a61193" containerName="payload-processing"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040383    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7442bea-8f57-4357-8a14-068130d772c6" containerName="maas-api"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040399    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040407    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc80910c-0735-4a9e-8d6f-1f902c0738cc" containerName="maas-api"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040418    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbc1604-9a77-4f6e-adea-2605b83d21c5" containerName="maas-api"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040429    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0ff4f5-ff1a-48f8-8746-3a4b49d4e5c3" containerName="payload-pre-processing"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040441    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8aba258-c5a6-4e67-9c6f-0037d6a61193" containerName="payload-processing"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040453    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="8082cf07-73a5-4469-9436-59d3d2f9eba1" containerName="payload-pre-processing"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040463    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040473    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy"
Oct 30 16:48:10.040566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.040482    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace03ed4-7a0d-49c2-bdf4-8d75e3eef9ce" containerName="istio-proxy"
Oct 30 16:48:10.046920 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.046893    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.066099 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.066068    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-cfe9f0-openshift-default-dockercfg-964lz\""
Oct 30 16:48:10.078767 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.078730    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"]
Oct 30 16:48:10.087968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.087920    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-certs\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.087968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.087966    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-token\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.088217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.088000    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-envoy\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.088217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.088160    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13fde00-4ff5-4167-84fc-5856dad1970c-istiod-ca-cert\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.088347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.088281    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-data\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.088449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.088429    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wh8\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-kube-api-access-b9wh8\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.088511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.088462    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-credential-socket\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.088619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.088595    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-podinfo\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.088681 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.088660    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-socket\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.189983    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-podinfo\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190034    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-socket\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190115    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-certs\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190139    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-token\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190168    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-envoy\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190234    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13fde00-4ff5-4167-84fc-5856dad1970c-istiod-ca-cert\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190420 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190298    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-data\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190423    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wh8\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-kube-api-access-b9wh8\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190451    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-credential-socket\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190590    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-socket\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190614    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-certs\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190848    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-data\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190909 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190861    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-credential-socket\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.190983 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.190965    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13fde00-4ff5-4167-84fc-5856dad1970c-istiod-ca-cert\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.192525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.192496    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-envoy\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.192643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.192627    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-podinfo\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.216258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.216228    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-token\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.216974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.216949    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wh8\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-kube-api-access-b9wh8\") pod \"e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") " pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.369610 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.369514    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:10.694756 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.694579    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"]
Oct 30 16:48:10.697857 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:48:10.697831    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fde00_4ff5_4167_84fc_5856dad1970c.slice/crio-47604144eb1128adde5a0043028e60580dc2557fe237f007a313799c963c0f01 WatchSource:0}: Error finding container 47604144eb1128adde5a0043028e60580dc2557fe237f007a313799c963c0f01: Status 404 returned error can't find the container with id 47604144eb1128adde5a0043028e60580dc2557fe237f007a313799c963c0f01
Oct 30 16:48:10.700446 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.700415    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:10.700535 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.700480    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:10.700535 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.700508    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:10.720655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:10.720620    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" event={"ID":"c13fde00-4ff5-4167-84fc-5856dad1970c","Type":"ContainerStarted","Data":"47604144eb1128adde5a0043028e60580dc2557fe237f007a313799c963c0f01"}
Oct 30 16:48:11.735146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:11.735084    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" event={"ID":"c13fde00-4ff5-4167-84fc-5856dad1970c","Type":"ContainerStarted","Data":"a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b"}
Oct 30 16:48:11.790678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:11.790615    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" podStartSLOduration=1.790599703 podStartE2EDuration="1.790599703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:11.788809006 +0000 UTC m=+1521.642687289" watchObservedRunningTime="2026-10-30 16:48:11.790599703 +0000 UTC m=+1521.644477972"
Oct 30 16:48:12.370266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:12.370219    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:12.376041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:12.376016    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:12.740933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:12.740857    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:12.741926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:12.741907    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:48:21.916980 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:21.916941    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:48:24.252952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.252912    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"]
Oct 30 16:48:24.253620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.253602    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy"
Oct 30 16:48:24.253668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.253621    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f1d7ae-9027-4ee7-8972-bc953e335c04" containerName="istio-proxy"
Oct 30 16:48:24.257268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.257250    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:24.280223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.279212    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2e9431-dockercfg-hczmp\""
Oct 30 16:48:24.280223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.279318    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2e9431\""
Oct 30 16:48:24.282758 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.282726    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"]
Oct 30 16:48:24.398672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.398594    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"]
Oct 30 16:48:24.418440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.418409    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"]
Oct 30 16:48:24.430207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.419613    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfwtm\" (UniqueName: \"kubernetes.io/projected/95a82982-3484-4b87-b94c-2217a310ffa8-kube-api-access-wfwtm\") pod \"payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7\" (UID: \"95a82982-3484-4b87-b94c-2217a310ffa8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:24.430686 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.430652    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a82982-3484-4b87-b94c-2217a310ffa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7\" (UID: \"95a82982-3484-4b87-b94c-2217a310ffa8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:24.443209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.431132    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:24.532725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.532684    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhkd\" (UniqueName: \"kubernetes.io/projected/3179bbae-5e62-48e2-a016-9f9da83dcb39-kube-api-access-gfhkd\") pod \"payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n\" (UID: \"3179bbae-5e62-48e2-a016-9f9da83dcb39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:24.532914 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.532800    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfwtm\" (UniqueName: \"kubernetes.io/projected/95a82982-3484-4b87-b94c-2217a310ffa8-kube-api-access-wfwtm\") pod \"payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7\" (UID: \"95a82982-3484-4b87-b94c-2217a310ffa8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:24.532914 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.532904    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a82982-3484-4b87-b94c-2217a310ffa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7\" (UID: \"95a82982-3484-4b87-b94c-2217a310ffa8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:24.533035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.532995    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3179bbae-5e62-48e2-a016-9f9da83dcb39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n\" (UID: \"3179bbae-5e62-48e2-a016-9f9da83dcb39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:24.533694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.533671    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a82982-3484-4b87-b94c-2217a310ffa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7\" (UID: \"95a82982-3484-4b87-b94c-2217a310ffa8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:24.554908 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.554663    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfwtm\" (UniqueName: \"kubernetes.io/projected/95a82982-3484-4b87-b94c-2217a310ffa8-kube-api-access-wfwtm\") pod \"payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7\" (UID: \"95a82982-3484-4b87-b94c-2217a310ffa8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:24.575527 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.575481    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:24.646282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.642585    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhkd\" (UniqueName: \"kubernetes.io/projected/3179bbae-5e62-48e2-a016-9f9da83dcb39-kube-api-access-gfhkd\") pod \"payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n\" (UID: \"3179bbae-5e62-48e2-a016-9f9da83dcb39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:24.646282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.642872    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3179bbae-5e62-48e2-a016-9f9da83dcb39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n\" (UID: \"3179bbae-5e62-48e2-a016-9f9da83dcb39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:24.646282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.643835    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3179bbae-5e62-48e2-a016-9f9da83dcb39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n\" (UID: \"3179bbae-5e62-48e2-a016-9f9da83dcb39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:24.680264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.679249    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhkd\" (UniqueName: \"kubernetes.io/projected/3179bbae-5e62-48e2-a016-9f9da83dcb39-kube-api-access-gfhkd\") pod \"payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n\" (UID: \"3179bbae-5e62-48e2-a016-9f9da83dcb39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:24.779558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.779513    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:24.954720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:24.954667    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"]
Oct 30 16:48:24.957782 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:48:24.957756    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a82982_3484_4b87_b94c_2217a310ffa8.slice/crio-6c260f84e476cc6811ab929ccd4831cbdfc6a03b2996ab145cfb6b337e7a5e85 WatchSource:0}: Error finding container 6c260f84e476cc6811ab929ccd4831cbdfc6a03b2996ab145cfb6b337e7a5e85: Status 404 returned error can't find the container with id 6c260f84e476cc6811ab929ccd4831cbdfc6a03b2996ab145cfb6b337e7a5e85
Oct 30 16:48:25.086373 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.086345    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"]
Oct 30 16:48:25.088046 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:48:25.088011    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3179bbae_5e62_48e2_a016_9f9da83dcb39.slice/crio-7be1a4376423a85057ea62f33cdf06b4ab1a5d87a0e9350ee22e15579fe27fb8 WatchSource:0}: Error finding container 7be1a4376423a85057ea62f33cdf06b4ab1a5d87a0e9350ee22e15579fe27fb8: Status 404 returned error can't find the container with id 7be1a4376423a85057ea62f33cdf06b4ab1a5d87a0e9350ee22e15579fe27fb8
Oct 30 16:48:25.831892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.831858    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n" event={"ID":"3179bbae-5e62-48e2-a016-9f9da83dcb39","Type":"ContainerStarted","Data":"8b0537ced2ea4d718c12503ab13c517b24254030871858e87c71b26d39528aa9"}
Oct 30 16:48:25.832444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.831915    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n" event={"ID":"3179bbae-5e62-48e2-a016-9f9da83dcb39","Type":"ContainerStarted","Data":"7be1a4376423a85057ea62f33cdf06b4ab1a5d87a0e9350ee22e15579fe27fb8"}
Oct 30 16:48:25.832444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.831946    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:25.833596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.833574    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7" event={"ID":"95a82982-3484-4b87-b94c-2217a310ffa8","Type":"ContainerStarted","Data":"ab47fff07a4d36033c95d2e549254bde66396f747471d778b342ef91f7d0dd18"}
Oct 30 16:48:25.833692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.833613    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7" event={"ID":"95a82982-3484-4b87-b94c-2217a310ffa8","Type":"ContainerStarted","Data":"6c260f84e476cc6811ab929ccd4831cbdfc6a03b2996ab145cfb6b337e7a5e85"}
Oct 30 16:48:25.833692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.833637    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:25.871086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.871036    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n" podStartSLOduration=1.8710206820000002 podStartE2EDuration="1.871020682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:25.867614702 +0000 UTC m=+1535.721492967" watchObservedRunningTime="2026-10-30 16:48:25.871020682 +0000 UTC m=+1535.724898944"
Oct 30 16:48:25.905938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:25.905891    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7" podStartSLOduration=1.9058769230000001 podStartE2EDuration="1.905876923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:25.905134719 +0000 UTC m=+1535.759012983" watchObservedRunningTime="2026-10-30 16:48:25.905876923 +0000 UTC m=+1535.759755199"
Oct 30 16:48:27.995607 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:27.995569    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"]
Oct 30 16:48:28.002217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.002159    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.008514 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.008480    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"]
Oct 30 16:48:28.014103 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.014077    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.022025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.021926    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ea8dca-openshift-default-dockercfg-79z6f\""
Oct 30 16:48:28.032442 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.032397    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"]
Oct 30 16:48:28.036938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.036894    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"]
Oct 30 16:48:28.054051 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.054018    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-504442-openshift-default-dockercfg-pvqxp\""
Oct 30 16:48:28.092460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092418    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istiod-ca-cert\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.092642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092469    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-data\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.092642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092497    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f80b1e-3475-477f-a6f3-d0069fedd05d-istiod-ca-cert\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.092642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092601    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpwql\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-kube-api-access-zpwql\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.092642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092627    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-certs\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.092783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092655    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-envoy\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.092783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092753    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-socket\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.092783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092779    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-token\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.092878 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092843    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-envoy\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.092911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.092880    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-token\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.093017 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.093000    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-podinfo\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.093054 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.093037    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-podinfo\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.093139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.093127    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-data\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.093179 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.093150    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-certs\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.093179 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.093165    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-credential-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.093303 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.093291    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.093344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.093312    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8drh\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-kube-api-access-n8drh\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.093378 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.093346    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-credential-socket\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.202207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202049    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpwql\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-kube-api-access-zpwql\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.202207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202108    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-certs\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.202207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202148    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-envoy\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.202207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202214    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-socket\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.202617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202242    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-token\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.202617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202306    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-envoy\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.202617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202357    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-token\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.202617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202453    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-podinfo\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.202617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202485    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-podinfo\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.202617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202555    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-data\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.202617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202586    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-certs\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.202617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202612    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-credential-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.203006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202684    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.203006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202712    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8drh\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-kube-api-access-n8drh\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.203006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202768    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-credential-socket\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.203006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202813    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istiod-ca-cert\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.203006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202843    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-data\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.203006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.202867    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f80b1e-3475-477f-a6f3-d0069fedd05d-istiod-ca-cert\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.207206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.203738    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f80b1e-3475-477f-a6f3-d0069fedd05d-istiod-ca-cert\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.207206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.204368    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-certs\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.218326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.217644    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-data\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.218326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.217919    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-socket\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.219409 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.219384    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-envoy\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.219816 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.219793    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-certs\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.220332 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.220298    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-credential-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.220543 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.220526    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.221751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.221022    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-credential-socket\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.221751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.221666    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istiod-ca-cert\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.221937 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.221920    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-data\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.239440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.234472    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-envoy\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.261115 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.261081    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpwql\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-kube-api-access-zpwql\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.267689 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.263365    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-podinfo\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.267689 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.263885    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-podinfo\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.269174 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.269106    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-token\") pod \"e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") " pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.270810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.270781    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8drh\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-kube-api-access-n8drh\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.271220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.271146    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-token\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.319767 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.319724    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:28.329125 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.329037    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:28.687037 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.686958    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"]
Oct 30 16:48:28.690147 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.690116    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"]
Oct 30 16:48:28.690346 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:48:28.690324    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb213dc75_c4b2_4ff1_8735_5d4d614b32d2.slice/crio-91a4d1586d2abd152370d89574e31b39f4d213c6558f510aa13d2f1a76e62e52 WatchSource:0}: Error finding container 91a4d1586d2abd152370d89574e31b39f4d213c6558f510aa13d2f1a76e62e52: Status 404 returned error can't find the container with id 91a4d1586d2abd152370d89574e31b39f4d213c6558f510aa13d2f1a76e62e52
Oct 30 16:48:28.693693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.693657    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:28.693816 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.693737    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:28.693816 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.693782    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:28.717838 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:48:28.717791    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f80b1e_3475_477f_a6f3_d0069fedd05d.slice/crio-237bf2d459a552f6a7967f8058946578b50632d03a8338a1a9749d1630f49b11 WatchSource:0}: Error finding container 237bf2d459a552f6a7967f8058946578b50632d03a8338a1a9749d1630f49b11: Status 404 returned error can't find the container with id 237bf2d459a552f6a7967f8058946578b50632d03a8338a1a9749d1630f49b11
Oct 30 16:48:28.719944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.719901    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:28.720042 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.720003    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:28.720104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.720046    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:48:28.910075 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.907127    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" event={"ID":"b213dc75-c4b2-4ff1-8735-5d4d614b32d2","Type":"ContainerStarted","Data":"d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336"}
Oct 30 16:48:28.910075 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.907232    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" event={"ID":"b213dc75-c4b2-4ff1-8735-5d4d614b32d2","Type":"ContainerStarted","Data":"91a4d1586d2abd152370d89574e31b39f4d213c6558f510aa13d2f1a76e62e52"}
Oct 30 16:48:28.910556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.910532    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" event={"ID":"55f80b1e-3475-477f-a6f3-d0069fedd05d","Type":"ContainerStarted","Data":"1bc18da8ff8b3934d2555578ab3c4bfdbbd57ecdd0d79f990e9c6c3f14cc5404"}
Oct 30 16:48:28.910661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:28.910586    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" event={"ID":"55f80b1e-3475-477f-a6f3-d0069fedd05d","Type":"ContainerStarted","Data":"237bf2d459a552f6a7967f8058946578b50632d03a8338a1a9749d1630f49b11"}
Oct 30 16:48:29.022423 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.022370    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podStartSLOduration=2.0223544430000002 podStartE2EDuration="2.022354443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:29.020399244 +0000 UTC m=+1538.874277508" watchObservedRunningTime="2026-10-30 16:48:29.022354443 +0000 UTC m=+1538.876232707"
Oct 30 16:48:29.024040 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.023991    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" podStartSLOduration=2.023974587 podStartE2EDuration="2.023974587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:28.952903215 +0000 UTC m=+1538.806781497" watchObservedRunningTime="2026-10-30 16:48:29.023974587 +0000 UTC m=+1538.877852854"
Oct 30 16:48:29.320768 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.320671    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:29.326921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.326891    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:29.330027 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.330001    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:29.331737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.331711    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 16:48:29.331864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.331757    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 16:48:29.411971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.411678    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"]
Oct 30 16:48:29.412821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.412776    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy" containerID="cri-o://c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700" gracePeriod=30
Oct 30 16:48:29.452630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.452602    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"]
Oct 30 16:48:29.453003 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.452973    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy" containerID="cri-o://5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5" gracePeriod=30
Oct 30 16:48:29.457312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.457283    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"]
Oct 30 16:48:29.457610 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.457586    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy" containerID="cri-o://80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9" gracePeriod=30
Oct 30 16:48:29.916476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.916445    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:29.917289 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:29.917268    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:48:30.330636 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:30.330590    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 16:48:30.331023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:30.330662    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 16:48:31.330094 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:31.330062    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 16:48:31.330267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:31.330115    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 16:48:32.330519 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:32.330488    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 16:48:32.330959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:32.330542    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 16:48:33.330293 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:33.330251    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 16:48:33.330495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:33.330318    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 16:48:34.329504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.329471    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 16:48:34.329903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.329526    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 16:48:34.821589 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.821561    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:48:34.853334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853252    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-data\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.853334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853314    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-socket\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.853530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853339    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-certs\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.853530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853467    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-token\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-token\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.853530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853501    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-podinfo\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.853628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853546    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-data" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.853628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853562    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-kube-api-access-crbjc\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-kube-api-access-crbjc\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.853697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853626    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-socket" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.853697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853681    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-envoy\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.853758 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853719    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-credential-socket\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.853758 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.853741    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37dd2c72-4998-4903-a89b-6c75511aaf65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37dd2c72-4998-4903-a89b-6c75511aaf65-istiod-ca-cert\") pod \"37dd2c72-4998-4903-a89b-6c75511aaf65\" (UID: \"37dd2c72-4998-4903-a89b-6c75511aaf65\") "
Oct 30 16:48:34.854223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.854173    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-certs" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.854223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.854206    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-credential-socket" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.854432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.854389    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.854432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.854403    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.854432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.854412    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.854432 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.854425    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.854639 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.854508    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37dd2c72-4998-4903-a89b-6c75511aaf65-istiod-ca-cert" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:34.856299 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.856268    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-podinfo" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:48:34.857414 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.857388    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-token" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:34.857523 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.857504    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-kube-api-access-crbjc" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "kube-api-access-crbjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:34.859101 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.859065    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-envoy" pod "37dd2c72-4998-4903-a89b-6c75511aaf65" (UID: "37dd2c72-4998-4903-a89b-6c75511aaf65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.910641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.910610    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:48:34.912547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.912525    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:48:34.951629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.951591    4497 generic.go:356] "Generic (PLEG): container finished" podID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerID="80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9" exitCode=0
Oct 30 16:48:34.951820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.951639    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"
Oct 30 16:48:34.951820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.951674    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" event={"ID":"0b58f985-587c-4614-9dde-96394ccf0cc2","Type":"ContainerDied","Data":"80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9"}
Oct 30 16:48:34.951820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.951731    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs" event={"ID":"0b58f985-587c-4614-9dde-96394ccf0cc2","Type":"ContainerDied","Data":"7dbd58b010d5b8b8aea16bc8c0fc92a2f1d254ff61f7345d1d46595ce0df7237"}
Oct 30 16:48:34.951820 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.951747    4497 scope.go:122] "RemoveContainer" containerID="80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9"
Oct 30 16:48:34.953283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.953257    4497 generic.go:356] "Generic (PLEG): container finished" podID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerID="5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5" exitCode=0
Oct 30 16:48:34.953283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.953292    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" event={"ID":"b8bdffc5-6351-41e4-86dd-f7e438c4be4e","Type":"ContainerDied","Data":"5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5"}
Oct 30 16:48:34.953493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.953347    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z" event={"ID":"b8bdffc5-6351-41e4-86dd-f7e438c4be4e","Type":"ContainerDied","Data":"5003212922f2dfe2d25ae2cd064161a4ec82636b06e893c88306a5c0fcae7bb8"}
Oct 30 16:48:34.953493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.953304    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"
Oct 30 16:48:34.954855 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.954829    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-kube-api-access-k6vbx\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-kube-api-access-k6vbx\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.954976 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.954886    4497 generic.go:356] "Generic (PLEG): container finished" podID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerID="c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700" exitCode=0
Oct 30 16:48:34.954976 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.954905    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-token\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-token\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.954976 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.954942    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-kube-api-access-mbppm\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-kube-api-access-mbppm\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.955144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.954981    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" event={"ID":"37dd2c72-4998-4903-a89b-6c75511aaf65","Type":"ContainerDied","Data":"c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700"}
Oct 30 16:48:34.955144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955016    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-data\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.955144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955070    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"
Oct 30 16:48:34.955144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955091    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-certs\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.955355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955144    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b58f985-587c-4614-9dde-96394ccf0cc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b58f985-587c-4614-9dde-96394ccf0cc2-istiod-ca-cert\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.955355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955206    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-podinfo\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.955355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955238    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-credential-socket\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.955355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955275    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-certs\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.955355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955340    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-socket\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.955609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955376    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-credential-socket\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.955609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955408    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-data\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.955609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955453    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-envoy\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.955609 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955486    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-socket\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.955821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955736    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-socket" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.956403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955942    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-credential-socket" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.956403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.955021    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj" event={"ID":"37dd2c72-4998-4903-a89b-6c75511aaf65","Type":"ContainerDied","Data":"cd5f899ef6d4b9fca9dc2dc0e76b1d90c898597ddb566d5d8cd05213501eb944"}
Oct 30 16:48:34.956403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.956133    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-data" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.956403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.956352    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-certs" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.956403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.956357    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istiod-ca-cert\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.956916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.956453    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-token\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-token\") pod \"0b58f985-587c-4614-9dde-96394ccf0cc2\" (UID: \"0b58f985-587c-4614-9dde-96394ccf0cc2\") "
Oct 30 16:48:34.956916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.956527    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b58f985-587c-4614-9dde-96394ccf0cc2-istiod-ca-cert" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:34.956916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.956604    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-podinfo\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.956916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.956804    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-envoy\") pod \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\" (UID: \"b8bdffc5-6351-41e4-86dd-f7e438c4be4e\") "
Oct 30 16:48:34.957134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.956930    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istiod-ca-cert" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957268    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-credential-socket" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957537    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957557    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37dd2c72-4998-4903-a89b-6c75511aaf65-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957571    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957583    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b58f985-587c-4614-9dde-96394ccf0cc2-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957597    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957610    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957624    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957638    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957651    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957667    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957683    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37dd2c72-4998-4903-a89b-6c75511aaf65-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.957723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.957697    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crbjc\" (UniqueName: \"kubernetes.io/projected/37dd2c72-4998-4903-a89b-6c75511aaf65-kube-api-access-crbjc\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:34.959067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.958987    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-data" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.959067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.959051    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-socket" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.959313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.959275    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-certs" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.963433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.963335    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-kube-api-access-k6vbx" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "kube-api-access-k6vbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:34.963433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.963392    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-podinfo" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:48:34.963930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.963889    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-kube-api-access-mbppm" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "kube-api-access-mbppm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:34.964148 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.964111    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-token" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:34.964372 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.964340    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-token" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:34.966072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.966051    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-podinfo" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:48:34.966664 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.966630    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-envoy" pod "0b58f985-587c-4614-9dde-96394ccf0cc2" (UID: "0b58f985-587c-4614-9dde-96394ccf0cc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.966764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.966660    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-envoy" pod "b8bdffc5-6351-41e4-86dd-f7e438c4be4e" (UID: "b8bdffc5-6351-41e4-86dd-f7e438c4be4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:34.968828 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.968809    4497 scope.go:122] "RemoveContainer" containerID="80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9"
Oct 30 16:48:34.969305 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:34.969276    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9\": container with ID starting with 80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9 not found: ID does not exist" containerID="80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9"
Oct 30 16:48:34.969397 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.969306    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9"} err="failed to get container status \"80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9\": rpc error: code = NotFound desc = could not find container \"80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9\": container with ID starting with 80c98431f01c420b3564f244be814d3a5f01953ca70efee68d9acd105919dcc9 not found: ID does not exist"
Oct 30 16:48:34.969397 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.969326    4497 scope.go:122] "RemoveContainer" containerID="5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5"
Oct 30 16:48:34.989088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.989061    4497 scope.go:122] "RemoveContainer" containerID="5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5"
Oct 30 16:48:34.989493 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:34.989465    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5\": container with ID starting with 5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5 not found: ID does not exist" containerID="5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5"
Oct 30 16:48:34.989581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.989507    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5"} err="failed to get container status \"5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5\": rpc error: code = NotFound desc = could not find container \"5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5\": container with ID starting with 5de3b4d4ccf5d9811e974c9a26718c336ab7209913067af65dad0fb025d57ba5 not found: ID does not exist"
Oct 30 16:48:34.989581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:34.989533    4497 scope.go:122] "RemoveContainer" containerID="c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700"
Oct 30 16:48:35.000089 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.000066    4497 scope.go:122] "RemoveContainer" containerID="c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700"
Oct 30 16:48:35.000513 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:35.000488    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700\": container with ID starting with c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700 not found: ID does not exist" containerID="c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700"
Oct 30 16:48:35.000614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.000520    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700"} err="failed to get container status \"c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700\": rpc error: code = NotFound desc = could not find container \"c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700\": container with ID starting with c4bfb7e721bfcfcf8067dc7860581c936e05b132c48581c54984877d176d9700 not found: ID does not exist"
Oct 30 16:48:35.059486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059452    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059478    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059486 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059487    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059495    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059505    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vbx\" (UniqueName: \"kubernetes.io/projected/0b58f985-587c-4614-9dde-96394ccf0cc2-kube-api-access-k6vbx\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059514    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059524    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbppm\" (UniqueName: \"kubernetes.io/projected/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-kube-api-access-mbppm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059532    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059540    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b58f985-587c-4614-9dde-96394ccf0cc2-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059549    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8bdffc5-6351-41e4-86dd-f7e438c4be4e-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.059723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.059556    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b58f985-587c-4614-9dde-96394ccf0cc2-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:35.066266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.066169    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"]
Oct 30 16:48:35.071745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.071714    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fr5gj"]
Oct 30 16:48:35.300852 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.300816    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"]
Oct 30 16:48:35.305735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.305699    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-pkh5z"]
Oct 30 16:48:35.344319 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.342012    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"]
Oct 30 16:48:35.358221 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.357035    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6v4bs"]
Oct 30 16:48:35.492485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.492452    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:35.493016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.492979    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:35.494391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:35.494368    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:48:36.300913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:36.300878    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" path="/var/lib/kubelet/pods/0b58f985-587c-4614-9dde-96394ccf0cc2/volumes"
Oct 30 16:48:36.301434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:36.301418    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" path="/var/lib/kubelet/pods/37dd2c72-4998-4903-a89b-6c75511aaf65/volumes"
Oct 30 16:48:36.301879 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:36.301866    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" path="/var/lib/kubelet/pods/b8bdffc5-6351-41e4-86dd-f7e438c4be4e/volumes"
Oct 30 16:48:36.843544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:36.843511    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:48:36.846207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:36.846015    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:48:38.648406 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.648374    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"]
Oct 30 16:48:38.648944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.648928    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy"
Oct 30 16:48:38.648944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.648944    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy"
Oct 30 16:48:38.649021 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.648967    4497 cpu_manager.go:414] "containerMap: removing container" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy"
Oct 30 16:48:38.649021 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.648972    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy"
Oct 30 16:48:38.649021 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.648980    4497 cpu_manager.go:414] "containerMap: removing container" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy"
Oct 30 16:48:38.649021 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.648986    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy"
Oct 30 16:48:38.649153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.649091    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8bdffc5-6351-41e4-86dd-f7e438c4be4e" containerName="istio-proxy"
Oct 30 16:48:38.649153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.649101    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="37dd2c72-4998-4903-a89b-6c75511aaf65" containerName="istio-proxy"
Oct 30 16:48:38.649153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.649106    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b58f985-587c-4614-9dde-96394ccf0cc2" containerName="istio-proxy"
Oct 30 16:48:38.654314 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.654293    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.682360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.682323    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cfe9f0-serving-cert\""
Oct 30 16:48:38.684070 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.684036    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"]
Oct 30 16:48:38.707057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.706999    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtz5\" (UniqueName: \"kubernetes.io/projected/824c439f-96b2-42d9-bed2-ff36ae080b2c-kube-api-access-dgtz5\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.707278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.707107    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.707278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.707254    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.707278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.707276    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824c439f-96b2-42d9-bed2-ff36ae080b2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.808332 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.808301    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtz5\" (UniqueName: \"kubernetes.io/projected/824c439f-96b2-42d9-bed2-ff36ae080b2c-kube-api-access-dgtz5\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.808499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.808405    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.808587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.808572    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.808627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.808606    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824c439f-96b2-42d9-bed2-ff36ae080b2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.809332 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.809302    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824c439f-96b2-42d9-bed2-ff36ae080b2c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.810860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.810832    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-metrics-certs\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.810952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.810879    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.832350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.832320    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtz5\" (UniqueName: \"kubernetes.io/projected/824c439f-96b2-42d9-bed2-ff36ae080b2c-kube-api-access-dgtz5\") pod \"maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:38.968285 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:38.968175    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:39.300690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:39.300657    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"]
Oct 30 16:48:39.304013 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:48:39.303988    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824c439f_96b2_42d9_bed2_ff36ae080b2c.slice/crio-8274c52d6a52a6be2c10279552cc17d58ad3230a37169157ae53df7a9fdad3ce WatchSource:0}: Error finding container 8274c52d6a52a6be2c10279552cc17d58ad3230a37169157ae53df7a9fdad3ce: Status 404 returned error can't find the container with id 8274c52d6a52a6be2c10279552cc17d58ad3230a37169157ae53df7a9fdad3ce
Oct 30 16:48:40.017974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:40.017939    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d" event={"ID":"824c439f-96b2-42d9-bed2-ff36ae080b2c","Type":"ContainerStarted","Data":"61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee"}
Oct 30 16:48:40.018428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:40.017999    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:40.018428 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:40.018017    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d" event={"ID":"824c439f-96b2-42d9-bed2-ff36ae080b2c","Type":"ContainerStarted","Data":"8274c52d6a52a6be2c10279552cc17d58ad3230a37169157ae53df7a9fdad3ce"}
Oct 30 16:48:40.065868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:40.062762    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d" podStartSLOduration=1.7008927520000001 podStartE2EDuration="2.062737932s" totalImagesPullingTime="361.84518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:40.060693315 +0000 UTC m=+1549.914571611" watchObservedRunningTime="2026-10-30 16:48:40.062737932 +0000 UTC m=+1549.916616253"
Oct 30 16:48:46.030182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:46.030148    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:48:46.343916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:46.343832    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"]
Oct 30 16:48:46.344077 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:46.344035    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy" containerID="cri-o://01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886" gracePeriod=30
Oct 30 16:48:48.526720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.526683    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"]
Oct 30 16:48:48.531067 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.531047    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.543929 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.543909    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-2e9431-serving-cert\""
Oct 30 16:48:48.557028 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.556996    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"]
Oct 30 16:48:48.663998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.663962    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1db025-db48-455f-8d67-0ee26e3f65f7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.664207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.664027    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.664268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.664221    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5nrb\" (UniqueName: \"kubernetes.io/projected/2f1db025-db48-455f-8d67-0ee26e3f65f7-kube-api-access-g5nrb\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.664268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.664255    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-metrics-certs\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.766343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.766307    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1db025-db48-455f-8d67-0ee26e3f65f7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.766615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.766367    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.766615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.766490    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5nrb\" (UniqueName: \"kubernetes.io/projected/2f1db025-db48-455f-8d67-0ee26e3f65f7-kube-api-access-g5nrb\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.766615 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.766519    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-metrics-certs\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.767057 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.767029    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1db025-db48-455f-8d67-0ee26e3f65f7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.768918 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.768899    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.768995 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.768919    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-metrics-certs\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.804512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.804400    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5nrb\" (UniqueName: \"kubernetes.io/projected/2f1db025-db48-455f-8d67-0ee26e3f65f7-kube-api-access-g5nrb\") pod \"maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:48.844984 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:48.844954    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:49.154424 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:49.154347    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"]
Oct 30 16:48:49.157662 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:48:49.157637    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1db025_db48_455f_8d67_0ee26e3f65f7.slice/crio-0d6ba20e41324f38adff1542532a5c2d20dc6adbef18e3dbda989707a2c7a199 WatchSource:0}: Error finding container 0d6ba20e41324f38adff1542532a5c2d20dc6adbef18e3dbda989707a2c7a199: Status 404 returned error can't find the container with id 0d6ba20e41324f38adff1542532a5c2d20dc6adbef18e3dbda989707a2c7a199
Oct 30 16:48:50.079746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:50.079714    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl" event={"ID":"2f1db025-db48-455f-8d67-0ee26e3f65f7","Type":"ContainerStarted","Data":"d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635"}
Oct 30 16:48:50.080283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:50.079772    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl" event={"ID":"2f1db025-db48-455f-8d67-0ee26e3f65f7","Type":"ContainerStarted","Data":"0d6ba20e41324f38adff1542532a5c2d20dc6adbef18e3dbda989707a2c7a199"}
Oct 30 16:48:50.080283 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:50.079930    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:50.123390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:50.123336    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl" podStartSLOduration=1.7133231960000002 podStartE2EDuration="2.123320764s" totalImagesPullingTime="409.997568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:50.119379011 +0000 UTC m=+1559.973257276" watchObservedRunningTime="2026-10-30 16:48:50.123320764 +0000 UTC m=+1559.977199031"
Oct 30 16:48:51.630296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.630238    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:48:51.705499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705467    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-kube-api-access-gx7m2\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-kube-api-access-gx7m2\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.705687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705523    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3e3495-7f0c-4435-b244-de4fa133b035-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3495-7f0c-4435-b244-de4fa133b035-istiod-ca-cert\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.705687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705577    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-podinfo\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.705687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705645    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-certs\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.705859 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705696    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-credential-socket\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.705859 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705736    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-token\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-token\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.705859 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705784    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-socket\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.705859 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705819    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-data\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.706060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.705908    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-envoy\") pod \"7f3e3495-7f0c-4435-b244-de4fa133b035\" (UID: \"7f3e3495-7f0c-4435-b244-de4fa133b035\") "
Oct 30 16:48:51.706157 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.706127    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3e3495-7f0c-4435-b244-de4fa133b035-istiod-ca-cert" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:48:51.707033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.706659    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-credential-socket" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:51.707033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.706705    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3e3495-7f0c-4435-b244-de4fa133b035-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.707033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.706768    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-socket" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:51.707033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.706829    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-certs" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:51.707033 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.706873    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-data" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:51.708410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.708318    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-kube-api-access-gx7m2" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "kube-api-access-gx7m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:51.710236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.710177    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-envoy" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:48:51.710236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.710212    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-token" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:48:51.710376 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.710306    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-podinfo" pod "7f3e3495-7f0c-4435-b244-de4fa133b035" (UID: "7f3e3495-7f0c-4435-b244-de4fa133b035"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:48:51.807772 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.807737    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.807772 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.807769    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.807772 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.807778    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.808011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.807785    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx7m2\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-kube-api-access-gx7m2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.808011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.807794    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.808011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.807803    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.808011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.807811    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3e3495-7f0c-4435-b244-de4fa133b035-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.808011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:51.807819    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3e3495-7f0c-4435-b244-de4fa133b035-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:48:51.916630 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:51.916543    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:48:52.094569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.094519    4497 generic.go:356] "Generic (PLEG): container finished" podID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerID="01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886" exitCode=0
Oct 30 16:48:52.094745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.094574    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"
Oct 30 16:48:52.094745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.094604    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" event={"ID":"7f3e3495-7f0c-4435-b244-de4fa133b035","Type":"ContainerDied","Data":"01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886"}
Oct 30 16:48:52.094745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.094661    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn" event={"ID":"7f3e3495-7f0c-4435-b244-de4fa133b035","Type":"ContainerDied","Data":"7e280bfea07e51701c712e7aa2b9332ceefaf722e8e8cc04524b7327d5ac560c"}
Oct 30 16:48:52.094745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.094687    4497 scope.go:122] "RemoveContainer" containerID="01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886"
Oct 30 16:48:52.115514 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.115486    4497 scope.go:122] "RemoveContainer" containerID="01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886"
Oct 30 16:48:52.116800 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:52.116773    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886\": container with ID starting with 01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886 not found: ID does not exist" containerID="01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886"
Oct 30 16:48:52.116905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.116815    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886"} err="failed to get container status \"01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886\": rpc error: code = NotFound desc = could not find container \"01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886\": container with ID starting with 01fea7ac4a069339936ffdc945bc49189d33bbe3dd7ed2ddd32044c8801f3886 not found: ID does not exist"
Oct 30 16:48:52.142824 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.142791    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"]
Oct 30 16:48:52.146571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.146535    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-rflxn"]
Oct 30 16:48:52.301130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:52.301091    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" path="/var/lib/kubelet/pods/7f3e3495-7f0c-4435-b244-de4fa133b035/volumes"
Oct 30 16:48:56.092274 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:56.092241    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:48:57.848129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.848094    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"]
Oct 30 16:48:57.848696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.848680    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy"
Oct 30 16:48:57.848740 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.848696    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy"
Oct 30 16:48:57.848818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.848808    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3e3495-7f0c-4435-b244-de4fa133b035" containerName="istio-proxy"
Oct 30 16:48:57.853503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.853485    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:57.865438 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.865408    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ea8dca-serving-cert\""
Oct 30 16:48:57.878533 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.878496    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"]
Oct 30 16:48:57.899231 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.899158    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce395c5c-0da1-4972-bb6a-0fba209464a9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:57.899400 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.899284    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:57.899400 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.899379    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:57.899538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:57.899521    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6th2\" (UniqueName: \"kubernetes.io/projected/ce395c5c-0da1-4972-bb6a-0fba209464a9-kube-api-access-r6th2\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.000838 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.000794    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce395c5c-0da1-4972-bb6a-0fba209464a9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.001044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.000850    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.001044 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.000944    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.001166 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:58.001070    4497 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-serving-cert: secret "maas-api-e2e-praxis-b-ea8dca-serving-cert" not found
Oct 30 16:48:58.001166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.001101    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6th2\" (UniqueName: \"kubernetes.io/projected/ce395c5c-0da1-4972-bb6a-0fba209464a9-kube-api-access-r6th2\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.001166 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:48:58.001162    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls podName:ce395c5c-0da1-4972-bb6a-0fba209464a9 nodeName:}" failed. No retries permitted until 2026-10-30 16:48:58.501137684 +0000 UTC m=+1568.355015952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls") pod "maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l" (UID: "ce395c5c-0da1-4972-bb6a-0fba209464a9") : secret "maas-api-e2e-praxis-b-ea8dca-serving-cert" not found
Oct 30 16:48:58.001578 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.001560    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce395c5c-0da1-4972-bb6a-0fba209464a9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.003335 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.003316    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.024259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.024219    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6th2\" (UniqueName: \"kubernetes.io/projected/ce395c5c-0da1-4972-bb6a-0fba209464a9-kube-api-access-r6th2\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.507541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.507503    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.509970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.509943    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:58.767854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:58.767768    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:48:59.075266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:59.075174    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"]
Oct 30 16:48:59.078914 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:48:59.078886    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce395c5c_0da1_4972_bb6a_0fba209464a9.slice/crio-cfeac46b0405f60cf7fad0c47ee5668647979a5a47d8af2e5c918278f42c84bf WatchSource:0}: Error finding container cfeac46b0405f60cf7fad0c47ee5668647979a5a47d8af2e5c918278f42c84bf: Status 404 returned error can't find the container with id cfeac46b0405f60cf7fad0c47ee5668647979a5a47d8af2e5c918278f42c84bf
Oct 30 16:48:59.161016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:48:59.160980    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l" event={"ID":"ce395c5c-0da1-4972-bb6a-0fba209464a9","Type":"ContainerStarted","Data":"cfeac46b0405f60cf7fad0c47ee5668647979a5a47d8af2e5c918278f42c84bf"}
Oct 30 16:49:00.168824 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:00.168790    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l" event={"ID":"ce395c5c-0da1-4972-bb6a-0fba209464a9","Type":"ContainerStarted","Data":"98c0fcf8d4294eb8b1750568622e29e07883a055e9e53a0296a3f584ab918a14"}
Oct 30 16:49:00.169282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:00.168849    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:49:00.208224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:00.208160    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l" podStartSLOduration=2.838364206 podStartE2EDuration="3.208143672s" totalImagesPullingTime="369.779466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:00.207259124 +0000 UTC m=+1570.061137388" watchObservedRunningTime="2026-10-30 16:49:00.208143672 +0000 UTC m=+1570.062021939"
Oct 30 16:49:02.458116 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.457897    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"]
Oct 30 16:49:02.465323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.465293    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.499212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.489218    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c408f1-openshift-default-dockercfg-966xc\""
Oct 30 16:49:02.507247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.505951    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"]
Oct 30 16:49:02.568960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.568911    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-istio-token\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.569162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.569013    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6mq\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-kube-api-access-xl6mq\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.569162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.569117    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-certs\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.569353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.569214    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-credential-socket\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.569353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.569238    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6964757-3358-4563-814a-89ba36f850da-istiod-ca-cert\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.569353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.569284    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6964757-3358-4563-814a-89ba36f850da-istio-podinfo\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.569353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.569312    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-data\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.569589 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.569363    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-socket\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.569589 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.569387    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-envoy\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684100    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-credential-socket\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684156    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6964757-3358-4563-814a-89ba36f850da-istiod-ca-cert\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684236    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6964757-3358-4563-814a-89ba36f850da-istio-podinfo\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684278    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-data\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684355    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-socket\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684388    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-envoy\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684458    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-istio-token\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684535    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6mq\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-kube-api-access-xl6mq\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.684579    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-certs\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.687212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.685118    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-certs\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.697212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.689841    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-socket\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.697212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.690543    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6964757-3358-4563-814a-89ba36f850da-istiod-ca-cert\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.697212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.691838    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-data\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.703210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.701526    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-credential-socket\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.728259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.723038    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6mq\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-kube-api-access-xl6mq\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.728259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.723456    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-istio-token\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.745210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.741558    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-envoy\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.773106 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.772180    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6964757-3358-4563-814a-89ba36f850da-istio-podinfo\") pod \"e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") " pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:02.816059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:02.815956    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:03.200567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:03.200533    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"]
Oct 30 16:49:03.201506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:03.201474    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:03.201596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:03.201538    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:03.201596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:03.201568    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:03.209048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:03.209015    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" event={"ID":"d6964757-3358-4563-814a-89ba36f850da","Type":"ContainerStarted","Data":"166cbaf14c725f747449bf3616c426b68bf219a31234564d9986568a620efc5d"}
Oct 30 16:49:04.217640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:04.217602    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" event={"ID":"d6964757-3358-4563-814a-89ba36f850da","Type":"ContainerStarted","Data":"e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2"}
Oct 30 16:49:04.290214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:04.284456    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" podStartSLOduration=2.284436736 podStartE2EDuration="2.284436736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:04.267836759 +0000 UTC m=+1574.121715024" watchObservedRunningTime="2026-10-30 16:49:04.284436736 +0000 UTC m=+1574.138315002"
Oct 30 16:49:04.816812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:04.816776    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:04.822752 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:04.822725    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:05.224138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:05.224042    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:05.225019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:05.224998    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:49:06.200060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:06.200028    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:49:11.265103 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:11.265069    4497 generic.go:356] "Generic (PLEG): container finished" podID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerID="a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b" exitCode=0
Oct 30 16:49:11.265584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:11.265145    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" event={"ID":"c13fde00-4ff5-4167-84fc-5856dad1970c","Type":"ContainerDied","Data":"a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b"}
Oct 30 16:49:11.265635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:11.265616    4497 scope.go:122] "RemoveContainer" containerID="a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b"
Oct 30 16:49:11.267212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:11.267163    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:11.267281 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:11.267243    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:11.267317 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:11.267280    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:12.345259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:12.341064    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" event={"ID":"c13fde00-4ff5-4167-84fc-5856dad1970c","Type":"ContainerStarted","Data":"752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336"}
Oct 30 16:49:12.369827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:12.369790    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:49:12.370916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:12.370891    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 16:49:12.371032 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:12.370947    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 16:49:12.741691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:12.741606    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:49:13.370014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:13.369968    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 16:49:13.370526 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:13.370048    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 16:49:14.370495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:14.370460    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 16:49:14.370859 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:14.370511    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 16:49:15.370208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:15.370165    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 16:49:15.370379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:15.370240    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 16:49:16.370513 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:16.370472    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 16:49:16.370990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:16.370533    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 16:49:17.369948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:17.369909    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 16:49:17.370155 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:17.369982    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 16:49:18.375132 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:18.375102    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:49:18.376418 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:18.376385    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:49:21.916150 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:49:21.916116    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:49:29.598144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:29.598091    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"]
Oct 30 16:49:29.599127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:29.599089    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy" containerID="cri-o://e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5" gracePeriod=30
Oct 30 16:49:30.614323 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:30.614290    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"]
Oct 30 16:49:30.614777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:30.614525    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy" containerID="cri-o://4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634" gracePeriod=30
Oct 30 16:49:34.535656 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:34.535618    4497 generic.go:356] "Generic (PLEG): container finished" podID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerID="d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336" exitCode=0
Oct 30 16:49:34.536367 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:34.535689    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" event={"ID":"b213dc75-c4b2-4ff1-8735-5d4d614b32d2","Type":"ContainerDied","Data":"d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336"}
Oct 30 16:49:34.536367 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:34.536207    4497 scope.go:122] "RemoveContainer" containerID="d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336"
Oct 30 16:49:34.538217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:34.538169    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:34.538316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:34.538262    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:34.538316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:34.538302    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:49:34.885536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:34.885510    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:49:35.046888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.046856    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-data\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047083 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.046929    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-socket\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047083 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.046953    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-certs\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047083 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047006    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-credential-socket\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047083 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047067    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-envoy\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047130    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-podinfo\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047157    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-kube-api-access-z256k\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-kube-api-access-z256k\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047221    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-data" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:35.047329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047227    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-socket" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:35.047329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047255    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-token\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-token\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047347    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istiod-ca-cert\") pod \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\" (UID: \"bd6c7303-9e9a-4a77-870d-548e12b8dfdf\") "
Oct 30 16:49:35.047601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047454    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-credential-socket" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:35.047721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047644    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-certs" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:35.047826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.047810    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istiod-ca-cert" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:49:35.049851 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.049827    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.049971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.049854    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.049971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.049867    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.049971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.049880    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.049971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.049893    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.050321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.050295    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-podinfo" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:49:35.050443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.050354    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-envoy" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:35.050517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.050453    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-token" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:49:35.050578 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.050557    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-kube-api-access-z256k" pod "bd6c7303-9e9a-4a77-870d-548e12b8dfdf" (UID: "bd6c7303-9e9a-4a77-870d-548e12b8dfdf"). InnerVolumeSpecName "kube-api-access-z256k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:49:35.081352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.081310    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"]
Oct 30 16:49:35.081697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.081552    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy" containerID="cri-o://19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f" gracePeriod=30
Oct 30 16:49:35.151371 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.151287    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.151371 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.151314    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.151371 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.151324    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z256k\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-kube-api-access-z256k\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.151371 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.151333    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6c7303-9e9a-4a77-870d-548e12b8dfdf-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:35.544350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.544314    4497 generic.go:356] "Generic (PLEG): container finished" podID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerID="e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5" exitCode=0
Oct 30 16:49:35.544812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.544367    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"
Oct 30 16:49:35.544812 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.544408    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" event={"ID":"bd6c7303-9e9a-4a77-870d-548e12b8dfdf","Type":"ContainerDied","Data":"e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5"}
Oct 30 16:49:35.544937 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.544856    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4" event={"ID":"bd6c7303-9e9a-4a77-870d-548e12b8dfdf","Type":"ContainerDied","Data":"9adf47f5f78f44a9f316ab5a5fc29f0028fa7ea4fa0cce8b76946b6449b9e741"}
Oct 30 16:49:35.544937 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.544880    4497 scope.go:122] "RemoveContainer" containerID="e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5"
Oct 30 16:49:35.546858 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.546832    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" event={"ID":"b213dc75-c4b2-4ff1-8735-5d4d614b32d2","Type":"ContainerStarted","Data":"79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d"}
Oct 30 16:49:35.555638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.555623    4497 scope.go:122] "RemoveContainer" containerID="e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5"
Oct 30 16:49:35.555908 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:49:35.555892    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5\": container with ID starting with e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5 not found: ID does not exist" containerID="e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5"
Oct 30 16:49:35.555973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.555917    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5"} err="failed to get container status \"e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5\": rpc error: code = NotFound desc = could not find container \"e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5\": container with ID starting with e40f41d852dfb8836781351c66022a90e012f6ebcdc93ed77b26b26c2c4aa4d5 not found: ID does not exist"
Oct 30 16:49:35.635747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.635690    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"]
Oct 30 16:49:35.645602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.645570    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-zwrv4"]
Oct 30 16:49:35.943121 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:35.943096    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:49:36.003542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003508    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-kube-api-access-smkr2\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.003717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003571    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-envoy\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.003717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003640    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-token\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-token\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.003869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003749    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-certs\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.003869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003776    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-credential-socket\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.003869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003830    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-socket\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.004034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003905    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-podinfo\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.004034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003943    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-data\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.004034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003997    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istiod-ca-cert\") pod \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\" (UID: \"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7\") "
Oct 30 16:49:36.004034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.003995    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-certs" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:36.004265 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.004149    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-socket" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:36.004607 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.004581    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.004718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.004611    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.004778 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.004737    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-data" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:36.004842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.004779    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-credential-socket" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:36.004905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.004886    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istiod-ca-cert" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:49:36.006371 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.006345    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-token" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:49:36.006894 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.006870    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-envoy" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:36.007888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.007133    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-kube-api-access-smkr2" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "kube-api-access-smkr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:49:36.007888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.007282    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-podinfo" pod "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" (UID: "3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:49:36.105823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.105731    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.105823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.105761    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smkr2\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-kube-api-access-smkr2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.105823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.105771    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.105823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.105781    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.105823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.105789    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.105823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.105804    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.105823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.105813    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:36.318673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.318621    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" path="/var/lib/kubelet/pods/bd6c7303-9e9a-4a77-870d-548e12b8dfdf/volumes"
Oct 30 16:49:36.322452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.322421    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:49:36.322638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.322589    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 16:49:36.322708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.322655    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 16:49:36.555675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.555641    4497 generic.go:356] "Generic (PLEG): container finished" podID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerID="4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634" exitCode=0
Oct 30 16:49:36.556102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.555712    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"
Oct 30 16:49:36.556102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.555730    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" event={"ID":"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7","Type":"ContainerDied","Data":"4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634"}
Oct 30 16:49:36.556102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.555782    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96" event={"ID":"3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7","Type":"ContainerDied","Data":"9c84b5df3f67189f3203ddd203c8214d539af936fda6713b05cf4a2abebcbbb6"}
Oct 30 16:49:36.556102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.555808    4497 scope.go:122] "RemoveContainer" containerID="4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634"
Oct 30 16:49:36.565826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.565805    4497 scope.go:122] "RemoveContainer" containerID="4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634"
Oct 30 16:49:36.566133 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:49:36.566111    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634\": container with ID starting with 4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634 not found: ID does not exist" containerID="4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634"
Oct 30 16:49:36.566231 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.566145    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634"} err="failed to get container status \"4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634\": rpc error: code = NotFound desc = could not find container \"4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634\": container with ID starting with 4d5ad2af7ca50b4f885260757159a4ce3351b917b474bc1300fe2a46cd53c634 not found: ID does not exist"
Oct 30 16:49:36.618026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.617982    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"]
Oct 30 16:49:36.623135 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:36.623107    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-p5n96"]
Oct 30 16:49:37.321018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:37.320981    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 16:49:37.321246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:37.321047    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 16:49:38.305100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:38.305064    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" path="/var/lib/kubelet/pods/3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7/volumes"
Oct 30 16:49:38.321177 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:38.321140    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 16:49:38.321388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:38.321218    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 16:49:39.321259 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:39.321216    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 16:49:39.321682 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:39.321283    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 16:49:40.323506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.323469    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 16:49:40.324090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.323534    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 16:49:40.405291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.405260    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:49:40.509080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509045    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-podinfo\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509204    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-certs\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509266    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-envoy\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509290    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-data\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509384 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509314    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218be99c-35a8-4d96-9726-edcc8a16bb42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218be99c-35a8-4d96-9726-edcc8a16bb42-istiod-ca-cert\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509384 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509371    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-token\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-token\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509445 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509401    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-socket\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509445 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509436    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-credential-socket\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509455    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-kube-api-access-65gpk\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-kube-api-access-65gpk\") pod \"218be99c-35a8-4d96-9726-edcc8a16bb42\" (UID: \"218be99c-35a8-4d96-9726-edcc8a16bb42\") "
Oct 30 16:49:40.509755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509727    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-data" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:40.509893 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.509874    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.510657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.510573    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-socket" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:40.510785 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.510671    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-credential-socket" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:40.510785 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.510695    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218be99c-35a8-4d96-9726-edcc8a16bb42-istiod-ca-cert" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:49:40.511166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.511145    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-certs" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:40.512744 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.512706    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-podinfo" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:49:40.512744 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.512729    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-kube-api-access-65gpk" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "kube-api-access-65gpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:49:40.514090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.514051    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-envoy" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:49:40.514276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.514259    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-token" pod "218be99c-35a8-4d96-9726-edcc8a16bb42" (UID: "218be99c-35a8-4d96-9726-edcc8a16bb42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:49:40.611084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611005    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.611300 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611274    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/218be99c-35a8-4d96-9726-edcc8a16bb42-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.611391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611383    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.611452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611445    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.611511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611499    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.611596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611582    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65gpk\" (UniqueName: \"kubernetes.io/projected/218be99c-35a8-4d96-9726-edcc8a16bb42-kube-api-access-65gpk\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.611677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611668    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/218be99c-35a8-4d96-9726-edcc8a16bb42-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.611759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611749    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/218be99c-35a8-4d96-9726-edcc8a16bb42-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:40.611962 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.611939    4497 generic.go:356] "Generic (PLEG): container finished" podID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerID="19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f" exitCode=0
Oct 30 16:49:40.612053 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.612032    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" event={"ID":"218be99c-35a8-4d96-9726-edcc8a16bb42","Type":"ContainerDied","Data":"19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f"}
Oct 30 16:49:40.612126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.612012    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"
Oct 30 16:49:40.612182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.612130    4497 scope.go:122] "RemoveContainer" containerID="19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f"
Oct 30 16:49:40.612379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.612361    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl" event={"ID":"218be99c-35a8-4d96-9726-edcc8a16bb42","Type":"ContainerDied","Data":"42f7a0eadc0df0f7dc49965f72f7b4332b56488cf437fdb1b0f1bd1081aeec95"}
Oct 30 16:49:40.624134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.624115    4497 scope.go:122] "RemoveContainer" containerID="19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f"
Oct 30 16:49:40.624492 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:49:40.624470    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f\": container with ID starting with 19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f not found: ID does not exist" containerID="19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f"
Oct 30 16:49:40.624597 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.624498    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f"} err="failed to get container status \"19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f\": rpc error: code = NotFound desc = could not find container \"19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f\": container with ID starting with 19d7d2b95d615f78978f52b023ba67ffcb6ad80df6daf6159fc8e9ed8027119f not found: ID does not exist"
Oct 30 16:49:40.659457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.659426    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"]
Oct 30 16:49:40.663806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:40.663779    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-kf7tl"]
Oct 30 16:49:41.327346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:41.327317    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:49:41.327763 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:41.327745    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:49:41.328563 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:41.328541    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:49:42.300251 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:42.300210    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" path="/var/lib/kubelet/pods/218be99c-35a8-4d96-9726-edcc8a16bb42/volumes"
Oct 30 16:49:44.661410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.661373    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"]
Oct 30 16:49:44.661984 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.661969    4497 cpu_manager.go:414] "containerMap: removing container" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy"
Oct 30 16:49:44.662031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.661984    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy"
Oct 30 16:49:44.662031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.662007    4497 cpu_manager.go:414] "containerMap: removing container" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy"
Oct 30 16:49:44.662031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.662012    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy"
Oct 30 16:49:44.662031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.662023    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy"
Oct 30 16:49:44.662031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.662029    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy"
Oct 30 16:49:44.662201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.662117    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6c7303-9e9a-4a77-870d-548e12b8dfdf" containerName="istio-proxy"
Oct 30 16:49:44.662201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.662124    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edcb04e-c6ad-4b52-9b15-8ad571f7c7b7" containerName="istio-proxy"
Oct 30 16:49:44.662201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.662136    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="218be99c-35a8-4d96-9726-edcc8a16bb42" containerName="istio-proxy"
Oct 30 16:49:44.665691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.665671    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.679646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.679625    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6d5903-serving-cert\""
Oct 30 16:49:44.692724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.692693    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"]
Oct 30 16:49:44.771537 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.771500    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xdd\" (UniqueName: \"kubernetes.io/projected/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-kube-api-access-z7xdd\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.771737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.771545    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.771737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.771687    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.771864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.771799    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.873524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.873475    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xdd\" (UniqueName: \"kubernetes.io/projected/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-kube-api-access-z7xdd\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.874069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.873528    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.874069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.873629    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.874069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.873785    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.874738 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.874711    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.877447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.877426    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.877447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.877439    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.898940 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.897922    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xdd\" (UniqueName: \"kubernetes.io/projected/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-kube-api-access-z7xdd\") pod \"maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:44.978942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:44.978844    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:45.282674 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:45.282547    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"]
Oct 30 16:49:45.285920 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:49:45.285891    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb85f85_4cab_4b18_bff7_bc7bb13967aa.slice/crio-5b6d7f7e38fd39e26d87118db119600f7cd823121ed2d473563952478f466419 WatchSource:0}: Error finding container 5b6d7f7e38fd39e26d87118db119600f7cd823121ed2d473563952478f466419: Status 404 returned error can't find the container with id 5b6d7f7e38fd39e26d87118db119600f7cd823121ed2d473563952478f466419
Oct 30 16:49:45.647251 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:45.647216    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7" event={"ID":"bbb85f85-4cab-4b18-bff7-bc7bb13967aa","Type":"ContainerStarted","Data":"5b6d7f7e38fd39e26d87118db119600f7cd823121ed2d473563952478f466419"}
Oct 30 16:49:46.655160 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:46.655124    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7" event={"ID":"bbb85f85-4cab-4b18-bff7-bc7bb13967aa","Type":"ContainerStarted","Data":"0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be"}
Oct 30 16:49:46.655662 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:46.655248    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:46.692762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:46.692708    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7" podStartSLOduration=2.365716333 podStartE2EDuration="2.69269383s" totalImagesPullingTime="326.977497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:46.69195308 +0000 UTC m=+1616.545831344" watchObservedRunningTime="2026-10-30 16:49:46.69269383 +0000 UTC m=+1616.546572094"
Oct 30 16:49:51.917089 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:49:51.917048    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:49:52.668856 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:52.668826    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:49:58.026130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.026097    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"]
Oct 30 16:49:58.026613 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.026373    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj" podUID="a8a6167e-9755-42ad-9d58-becd7a81cf2c" containerName="maas-api" containerID="cri-o://3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3" gracePeriod=30
Oct 30 16:49:58.502961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.502931    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:49:58.569360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.569270    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-metrics-certs\") pod \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") "
Oct 30 16:49:58.569532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.569438    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a6167e-9755-42ad-9d58-becd7a81cf2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a6167e-9755-42ad-9d58-becd7a81cf2c-openshift-service-ca\") pod \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") "
Oct 30 16:49:58.569532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.569517    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a6167e-9755-42ad-9d58-becd7a81cf2c-kube-api-access-7vpcv\" (UniqueName: \"kubernetes.io/projected/a8a6167e-9755-42ad-9d58-becd7a81cf2c-kube-api-access-7vpcv\") pod \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") "
Oct 30 16:49:58.569663 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.569548    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-maas-api-tls\") pod \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\" (UID: \"a8a6167e-9755-42ad-9d58-becd7a81cf2c\") "
Oct 30 16:49:58.571766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.571732    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a6167e-9755-42ad-9d58-becd7a81cf2c-openshift-service-ca" pod "a8a6167e-9755-42ad-9d58-becd7a81cf2c" (UID: "a8a6167e-9755-42ad-9d58-becd7a81cf2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:49:58.573625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.573590    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-metrics-certs" pod "a8a6167e-9755-42ad-9d58-becd7a81cf2c" (UID: "a8a6167e-9755-42ad-9d58-becd7a81cf2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:49:58.574843 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.574806    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-maas-api-tls" pod "a8a6167e-9755-42ad-9d58-becd7a81cf2c" (UID: "a8a6167e-9755-42ad-9d58-becd7a81cf2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:49:58.575988 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.575952    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a6167e-9755-42ad-9d58-becd7a81cf2c-kube-api-access-7vpcv" pod "a8a6167e-9755-42ad-9d58-becd7a81cf2c" (UID: "a8a6167e-9755-42ad-9d58-becd7a81cf2c"). InnerVolumeSpecName "kube-api-access-7vpcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:49:58.670959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.670927    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:58.670959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.670955    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8a6167e-9755-42ad-9d58-becd7a81cf2c-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:58.670959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.670966    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vpcv\" (UniqueName: \"kubernetes.io/projected/a8a6167e-9755-42ad-9d58-becd7a81cf2c-kube-api-access-7vpcv\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:58.671205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.670975    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8a6167e-9755-42ad-9d58-becd7a81cf2c-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:49:58.749989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.749953    4497 generic.go:356] "Generic (PLEG): container finished" podID="a8a6167e-9755-42ad-9d58-becd7a81cf2c" containerID="3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3" exitCode=0
Oct 30 16:49:58.750208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.749993    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj" event={"ID":"a8a6167e-9755-42ad-9d58-becd7a81cf2c","Type":"ContainerDied","Data":"3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3"}
Oct 30 16:49:58.750208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.750034    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"
Oct 30 16:49:58.750208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.750058    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj" event={"ID":"a8a6167e-9755-42ad-9d58-becd7a81cf2c","Type":"ContainerDied","Data":"7a037f8956fa84af8098bd2a7804bc5d0bcc665d061d92c727b612dbfb807542"}
Oct 30 16:49:58.750208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.750085    4497 scope.go:122] "RemoveContainer" containerID="3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3"
Oct 30 16:49:58.762172 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.762153    4497 scope.go:122] "RemoveContainer" containerID="3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3"
Oct 30 16:49:58.762494 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:49:58.762466    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3\": container with ID starting with 3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3 not found: ID does not exist" containerID="3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3"
Oct 30 16:49:58.762601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.762502    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3"} err="failed to get container status \"3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3\": rpc error: code = NotFound desc = could not find container \"3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3\": container with ID starting with 3f734f61ece003ba1c568092b81615de4a13ca6f4c446025e59c882a6c9121a3 not found: ID does not exist"
Oct 30 16:49:58.799113 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.799075    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"]
Oct 30 16:49:58.803886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:49:58.803858    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-504442-6bb94f47d9-t9ttj"]
Oct 30 16:50:00.300092 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:00.300052    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a6167e-9755-42ad-9d58-becd7a81cf2c" path="/var/lib/kubelet/pods/a8a6167e-9755-42ad-9d58-becd7a81cf2c/volumes"
Oct 30 16:50:09.949965 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:09.949920    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"]
Oct 30 16:50:09.950605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:09.950205    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4" podUID="2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" containerName="discovery" containerID="cri-o://1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370" gracePeriod=30
Oct 30 16:50:10.251001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.250975    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:50:10.368394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.368354    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-ca-configmap\") pod \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") "
Oct 30 16:50:10.368595 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.368490    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-kubeconfig\") pod \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") "
Oct 30 16:50:10.368595 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.368589    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-local-certs\") pod \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") "
Oct 30 16:50:10.368720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.368646    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-token\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-token\") pod \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") "
Oct 30 16:50:10.368720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.368700    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-cacerts\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-cacerts\") pod \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") "
Oct 30 16:50:10.368829 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.368733    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-ca-configmap" pod "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" (UID: "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:50:10.368892 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.368870    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-dns-cert\") pod \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") "
Oct 30 16:50:10.368953 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.368906    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-kube-api-access-m8ll2\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-kube-api-access-m8ll2\") pod \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\" (UID: \"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa\") "
Oct 30 16:50:10.369542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.369517    4497 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-ca-configmap\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:10.371687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.371658    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-kubeconfig" pod "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" (UID: "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:50:10.371797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.371756    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-local-certs" pod "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" (UID: "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:10.372261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.372231    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-cacerts" pod "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" (UID: "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:50:10.372352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.372315    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-dns-cert" pod "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" (UID: "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:50:10.372455 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.372439    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-token" pod "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" (UID: "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:10.372604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.372589    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-kube-api-access-m8ll2" pod "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" (UID: "2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa"). InnerVolumeSpecName "kube-api-access-m8ll2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:10.471066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.470929    4497 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-kubeconfig\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:10.471066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.470955    4497 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-local-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:10.471066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.470963    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:10.471066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.470973    4497 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-cacerts\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:10.471066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.470985    4497 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-istio-csr-dns-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:10.471066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.470999    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8ll2\" (UniqueName: \"kubernetes.io/projected/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa-kube-api-access-m8ll2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:10.840760 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.840726    4497 generic.go:356] "Generic (PLEG): container finished" podID="2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" containerID="1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370" exitCode=0
Oct 30 16:50:10.840957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.840774    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"
Oct 30 16:50:10.840957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.840818    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4" event={"ID":"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa","Type":"ContainerDied","Data":"1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370"}
Oct 30 16:50:10.840957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.840880    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4" event={"ID":"2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa","Type":"ContainerDied","Data":"5a8ed32710777d60922b5762be44d1fcdad59b99c3a0ec30a941e72743f36d18"}
Oct 30 16:50:10.840957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.840907    4497 scope.go:122] "RemoveContainer" containerID="1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370"
Oct 30 16:50:10.851642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.851626    4497 scope.go:122] "RemoveContainer" containerID="1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370"
Oct 30 16:50:10.851956 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:50:10.851936    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370\": container with ID starting with 1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370 not found: ID does not exist" containerID="1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370"
Oct 30 16:50:10.852016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.851964    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370"} err="failed to get container status \"1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370\": rpc error: code = NotFound desc = could not find container \"1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370\": container with ID starting with 1015da3071e4b4a2df07f8b709cc7dd2daaae6b365013000049d74db2d304370 not found: ID does not exist"
Oct 30 16:50:10.885348 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.885322    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"]
Oct 30 16:50:10.892338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:10.892310    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bwmh4"]
Oct 30 16:50:12.300521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:12.300482    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" path="/var/lib/kubelet/pods/2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa/volumes"
Oct 30 16:50:12.858559 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:12.858465    4497 generic.go:356] "Generic (PLEG): container finished" podID="d6964757-3358-4563-814a-89ba36f850da" containerID="e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2" exitCode=0
Oct 30 16:50:12.858769 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:12.858549    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" event={"ID":"d6964757-3358-4563-814a-89ba36f850da","Type":"ContainerDied","Data":"e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2"}
Oct 30 16:50:12.859112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:12.859090    4497 scope.go:122] "RemoveContainer" containerID="e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2"
Oct 30 16:50:12.861837 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:12.861808    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:50:12.861910 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:12.861887    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:50:12.861950 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:12.861928    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:50:13.867166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:13.867132    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" event={"ID":"d6964757-3358-4563-814a-89ba36f850da","Type":"ContainerStarted","Data":"66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491"}
Oct 30 16:50:14.816894 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:14.816863    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:50:14.817380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:14.817319    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 16:50:14.817915 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:14.817404    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 16:50:15.816524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:15.816491    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 16:50:15.816919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:15.816543    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 16:50:16.816612 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:16.816570    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 16:50:16.817127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:16.816646    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 16:50:17.817163 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:17.817126    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 16:50:17.817625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:17.817199    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 16:50:18.817137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:18.817095    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 16:50:18.817365 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:18.817152    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 16:50:19.821080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:19.821050    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:50:19.821562 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:19.821437    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:50:19.822311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:19.822289    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:50:21.916321 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:50:21.916284    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:50:22.838296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:22.838260    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"]
Oct 30 16:50:22.873683 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:22.873650    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"]
Oct 30 16:50:23.709607 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:23.709572    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"]
Oct 30 16:50:23.710157 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:23.709787    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l" podUID="ce395c5c-0da1-4972-bb6a-0fba209464a9" containerName="maas-api" containerID="cri-o://98c0fcf8d4294eb8b1750568622e29e07883a055e9e53a0296a3f584ab918a14" gracePeriod=30
Oct 30 16:50:23.985818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:23.985715    4497 generic.go:356] "Generic (PLEG): container finished" podID="ce395c5c-0da1-4972-bb6a-0fba209464a9" containerID="98c0fcf8d4294eb8b1750568622e29e07883a055e9e53a0296a3f584ab918a14" exitCode=0
Oct 30 16:50:23.986090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:23.985851    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l" event={"ID":"ce395c5c-0da1-4972-bb6a-0fba209464a9","Type":"ContainerDied","Data":"98c0fcf8d4294eb8b1750568622e29e07883a055e9e53a0296a3f584ab918a14"}
Oct 30 16:50:23.986090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:23.985904    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l" event={"ID":"ce395c5c-0da1-4972-bb6a-0fba209464a9","Type":"ContainerDied","Data":"cfeac46b0405f60cf7fad0c47ee5668647979a5a47d8af2e5c918278f42c84bf"}
Oct 30 16:50:23.986090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:23.985924    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfeac46b0405f60cf7fad0c47ee5668647979a5a47d8af2e5c918278f42c84bf"
Oct 30 16:50:23.992796 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:23.991809    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:50:24.096781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.096743    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce395c5c-0da1-4972-bb6a-0fba209464a9-kube-api-access-r6th2\" (UniqueName: \"kubernetes.io/projected/ce395c5c-0da1-4972-bb6a-0fba209464a9-kube-api-access-r6th2\") pod \"ce395c5c-0da1-4972-bb6a-0fba209464a9\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") "
Oct 30 16:50:24.096948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.096876    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-metrics-certs\") pod \"ce395c5c-0da1-4972-bb6a-0fba209464a9\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") "
Oct 30 16:50:24.097025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.096998    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls\") pod \"ce395c5c-0da1-4972-bb6a-0fba209464a9\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") "
Oct 30 16:50:24.097159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.097045    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce395c5c-0da1-4972-bb6a-0fba209464a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce395c5c-0da1-4972-bb6a-0fba209464a9-openshift-service-ca\") pod \"ce395c5c-0da1-4972-bb6a-0fba209464a9\" (UID: \"ce395c5c-0da1-4972-bb6a-0fba209464a9\") "
Oct 30 16:50:24.097493 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.097466    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce395c5c-0da1-4972-bb6a-0fba209464a9-openshift-service-ca" pod "ce395c5c-0da1-4972-bb6a-0fba209464a9" (UID: "ce395c5c-0da1-4972-bb6a-0fba209464a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:50:24.097695 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.097675    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce395c5c-0da1-4972-bb6a-0fba209464a9-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:24.099018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.098991    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls" pod "ce395c5c-0da1-4972-bb6a-0fba209464a9" (UID: "ce395c5c-0da1-4972-bb6a-0fba209464a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:50:24.099108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.099023    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-metrics-certs" pod "ce395c5c-0da1-4972-bb6a-0fba209464a9" (UID: "ce395c5c-0da1-4972-bb6a-0fba209464a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:50:24.099108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.099032    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce395c5c-0da1-4972-bb6a-0fba209464a9-kube-api-access-r6th2" pod "ce395c5c-0da1-4972-bb6a-0fba209464a9" (UID: "ce395c5c-0da1-4972-bb6a-0fba209464a9"). InnerVolumeSpecName "kube-api-access-r6th2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:24.202572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.202470    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6th2\" (UniqueName: \"kubernetes.io/projected/ce395c5c-0da1-4972-bb6a-0fba209464a9-kube-api-access-r6th2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:24.202572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.202510    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:24.202572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.202528    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce395c5c-0da1-4972-bb6a-0fba209464a9-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:24.999770 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:24.999737    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"
Oct 30 16:50:25.045445 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:25.045410    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"]
Oct 30 16:50:25.054263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:25.054235    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea8dca-55887ccf5c-5f99l"]
Oct 30 16:50:26.301402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:26.301367    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce395c5c-0da1-4972-bb6a-0fba209464a9" path="/var/lib/kubelet/pods/ce395c5c-0da1-4972-bb6a-0fba209464a9/volumes"
Oct 30 16:50:27.839075 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:27.839014    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n" podUID="3179bbae-5e62-48e2-a016-9f9da83dcb39" containerName="payload-processing" containerID="cri-o://8b0537ced2ea4d718c12503ab13c517b24254030871858e87c71b26d39528aa9" gracePeriod=25
Oct 30 16:50:27.874702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:27.874661    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7" podUID="95a82982-3484-4b87-b94c-2217a310ffa8" containerName="payload-pre-processing" containerID="cri-o://ab47fff07a4d36033c95d2e549254bde66396f747471d778b342ef91f7d0dd18" gracePeriod=25
Oct 30 16:50:28.021711 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.021676    4497 generic.go:356] "Generic (PLEG): container finished" podID="3179bbae-5e62-48e2-a016-9f9da83dcb39" containerID="8b0537ced2ea4d718c12503ab13c517b24254030871858e87c71b26d39528aa9" exitCode=0
Oct 30 16:50:28.021944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.021777    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n" event={"ID":"3179bbae-5e62-48e2-a016-9f9da83dcb39","Type":"ContainerDied","Data":"8b0537ced2ea4d718c12503ab13c517b24254030871858e87c71b26d39528aa9"}
Oct 30 16:50:28.023480 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.023450    4497 generic.go:356] "Generic (PLEG): container finished" podID="95a82982-3484-4b87-b94c-2217a310ffa8" containerID="ab47fff07a4d36033c95d2e549254bde66396f747471d778b342ef91f7d0dd18" exitCode=0
Oct 30 16:50:28.023625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.023539    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7" event={"ID":"95a82982-3484-4b87-b94c-2217a310ffa8","Type":"ContainerDied","Data":"ab47fff07a4d36033c95d2e549254bde66396f747471d778b342ef91f7d0dd18"}
Oct 30 16:50:28.147113 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.147084    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:50:28.185566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.185540    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:50:28.252264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.252233    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a82982-3484-4b87-b94c-2217a310ffa8-kube-api-access-wfwtm\" (UniqueName: \"kubernetes.io/projected/95a82982-3484-4b87-b94c-2217a310ffa8-kube-api-access-wfwtm\") pod \"95a82982-3484-4b87-b94c-2217a310ffa8\" (UID: \"95a82982-3484-4b87-b94c-2217a310ffa8\") "
Oct 30 16:50:28.252449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.252332    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3179bbae-5e62-48e2-a016-9f9da83dcb39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3179bbae-5e62-48e2-a016-9f9da83dcb39-plugins-config-volume\") pod \"3179bbae-5e62-48e2-a016-9f9da83dcb39\" (UID: \"3179bbae-5e62-48e2-a016-9f9da83dcb39\") "
Oct 30 16:50:28.252529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.252512    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a82982-3484-4b87-b94c-2217a310ffa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a82982-3484-4b87-b94c-2217a310ffa8-plugins-config-volume\") pod \"95a82982-3484-4b87-b94c-2217a310ffa8\" (UID: \"95a82982-3484-4b87-b94c-2217a310ffa8\") "
Oct 30 16:50:28.252680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.252644    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3179bbae-5e62-48e2-a016-9f9da83dcb39-kube-api-access-gfhkd\" (UniqueName: \"kubernetes.io/projected/3179bbae-5e62-48e2-a016-9f9da83dcb39-kube-api-access-gfhkd\") pod \"3179bbae-5e62-48e2-a016-9f9da83dcb39\" (UID: \"3179bbae-5e62-48e2-a016-9f9da83dcb39\") "
Oct 30 16:50:28.252905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.252877    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a82982-3484-4b87-b94c-2217a310ffa8-plugins-config-volume" pod "95a82982-3484-4b87-b94c-2217a310ffa8" (UID: "95a82982-3484-4b87-b94c-2217a310ffa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:50:28.253117 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.253085    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3179bbae-5e62-48e2-a016-9f9da83dcb39-plugins-config-volume" pod "3179bbae-5e62-48e2-a016-9f9da83dcb39" (UID: "3179bbae-5e62-48e2-a016-9f9da83dcb39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:50:28.253272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.253257    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a82982-3484-4b87-b94c-2217a310ffa8-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:28.253346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.253273    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3179bbae-5e62-48e2-a016-9f9da83dcb39-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:28.254889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.254862    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3179bbae-5e62-48e2-a016-9f9da83dcb39-kube-api-access-gfhkd" pod "3179bbae-5e62-48e2-a016-9f9da83dcb39" (UID: "3179bbae-5e62-48e2-a016-9f9da83dcb39"). InnerVolumeSpecName "kube-api-access-gfhkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:28.255008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.254934    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a82982-3484-4b87-b94c-2217a310ffa8-kube-api-access-wfwtm" pod "95a82982-3484-4b87-b94c-2217a310ffa8" (UID: "95a82982-3484-4b87-b94c-2217a310ffa8"). InnerVolumeSpecName "kube-api-access-wfwtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:28.354333 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.354239    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhkd\" (UniqueName: \"kubernetes.io/projected/3179bbae-5e62-48e2-a016-9f9da83dcb39-kube-api-access-gfhkd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:28.354333 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:28.354268    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfwtm\" (UniqueName: \"kubernetes.io/projected/95a82982-3484-4b87-b94c-2217a310ffa8-kube-api-access-wfwtm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:29.030375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.030341    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"
Oct 30 16:50:29.030375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.030351    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n" event={"ID":"3179bbae-5e62-48e2-a016-9f9da83dcb39","Type":"ContainerDied","Data":"7be1a4376423a85057ea62f33cdf06b4ab1a5d87a0e9350ee22e15579fe27fb8"}
Oct 30 16:50:29.030905 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.030417    4497 scope.go:122] "RemoveContainer" containerID="8b0537ced2ea4d718c12503ab13c517b24254030871858e87c71b26d39528aa9"
Oct 30 16:50:29.031897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.031880    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"
Oct 30 16:50:29.031987 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.031938    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7" event={"ID":"95a82982-3484-4b87-b94c-2217a310ffa8","Type":"ContainerDied","Data":"6c260f84e476cc6811ab929ccd4831cbdfc6a03b2996ab145cfb6b337e7a5e85"}
Oct 30 16:50:29.039979 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.039959    4497 scope.go:122] "RemoveContainer" containerID="ab47fff07a4d36033c95d2e549254bde66396f747471d778b342ef91f7d0dd18"
Oct 30 16:50:29.149431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.149345    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"]
Oct 30 16:50:29.173743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.173712    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2e9431-7d4d96fcf5-57lt7"]
Oct 30 16:50:29.202567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.202533    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"]
Oct 30 16:50:29.207293 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:29.207262    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2e9431-c6b54dfd6-6s28n"]
Oct 30 16:50:30.300265 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:30.300233    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3179bbae-5e62-48e2-a016-9f9da83dcb39" path="/var/lib/kubelet/pods/3179bbae-5e62-48e2-a016-9f9da83dcb39/volumes"
Oct 30 16:50:30.300654 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:30.300619    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a82982-3484-4b87-b94c-2217a310ffa8" path="/var/lib/kubelet/pods/95a82982-3484-4b87-b94c-2217a310ffa8/volumes"
Oct 30 16:50:32.013046 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:32.007347    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"]
Oct 30 16:50:32.013046 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:32.007672    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy" containerID="cri-o://79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d" gracePeriod=30
Oct 30 16:50:37.283448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.283383    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:50:37.391821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.391786    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-certs\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.391969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.391830    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-data\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.391969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.391873    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-kube-api-access-zpwql\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-kube-api-access-zpwql\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.391969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.391930    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-socket\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.392175 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392050    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-envoy\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.392175 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392105    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-certs" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:37.392175 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392129    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-podinfo\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.392383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392227    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-data" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:37.392383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392243    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-token\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-token\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.392383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392264    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-socket" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:37.392383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392367    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-credential-socket\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.392598 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392400    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istiod-ca-cert\") pod \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\" (UID: \"b213dc75-c4b2-4ff1-8735-5d4d614b32d2\") "
Oct 30 16:50:37.392650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392625    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-credential-socket" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:37.392945 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.392919    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istiod-ca-cert" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:50:37.393408 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.393142    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:37.393408 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.393167    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:37.393408 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.393180    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:37.393408 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.393233    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:37.393408 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.393247    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:37.394391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.394367    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-envoy" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:37.394496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.394477    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-kube-api-access-zpwql" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "kube-api-access-zpwql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:37.394556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.394488    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-podinfo" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:50:37.394611 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.394548    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-token" pod "b213dc75-c4b2-4ff1-8735-5d4d614b32d2" (UID: "b213dc75-c4b2-4ff1-8735-5d4d614b32d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:37.494434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.494400    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpwql\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-kube-api-access-zpwql\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:37.494434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.494429    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:37.494434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.494438    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:37.494434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:37.494447    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b213dc75-c4b2-4ff1-8735-5d4d614b32d2-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:38.094505 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.094469    4497 generic.go:356] "Generic (PLEG): container finished" podID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerID="79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d" exitCode=0
Oct 30 16:50:38.094696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.094538    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"
Oct 30 16:50:38.094696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.094552    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" event={"ID":"b213dc75-c4b2-4ff1-8735-5d4d614b32d2","Type":"ContainerDied","Data":"79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d"}
Oct 30 16:50:38.094696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.094630    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9" event={"ID":"b213dc75-c4b2-4ff1-8735-5d4d614b32d2","Type":"ContainerDied","Data":"91a4d1586d2abd152370d89574e31b39f4d213c6558f510aa13d2f1a76e62e52"}
Oct 30 16:50:38.094696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.094652    4497 scope.go:122] "RemoveContainer" containerID="79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d"
Oct 30 16:50:38.106035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.106010    4497 scope.go:122] "RemoveContainer" containerID="d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336"
Oct 30 16:50:38.114971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.114952    4497 scope.go:122] "RemoveContainer" containerID="79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d"
Oct 30 16:50:38.115265 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:50:38.115244    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d\": container with ID starting with 79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d not found: ID does not exist" containerID="79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d"
Oct 30 16:50:38.115350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.115272    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d"} err="failed to get container status \"79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d\": rpc error: code = NotFound desc = could not find container \"79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d\": container with ID starting with 79cc95e9e7b24af7a74660be72b2f6f0a22fd8e6ced2f537df1ebe56d18b528d not found: ID does not exist"
Oct 30 16:50:38.115350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.115290    4497 scope.go:122] "RemoveContainer" containerID="d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336"
Oct 30 16:50:38.115528 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:50:38.115511    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336\": container with ID starting with d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336 not found: ID does not exist" containerID="d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336"
Oct 30 16:50:38.115572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.115536    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336"} err="failed to get container status \"d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336\": rpc error: code = NotFound desc = could not find container \"d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336\": container with ID starting with d0854ed6863fa1506cc6f4192e8004ca805153a1ad0168cc0b9ca6924e2b8336 not found: ID does not exist"
Oct 30 16:50:38.140635 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.140602    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"]
Oct 30 16:50:38.150004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.149975    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ea8dca-openshift-default-76dc9dfc77-vrfq9"]
Oct 30 16:50:38.301720 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:38.301685    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" path="/var/lib/kubelet/pods/b213dc75-c4b2-4ff1-8735-5d4d614b32d2/volumes"
Oct 30 16:50:40.603804 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.603774    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"]
Oct 30 16:50:40.604457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.603984    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl" podUID="2f1db025-db48-455f-8d67-0ee26e3f65f7" containerName="maas-api" containerID="cri-o://d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635" gracePeriod=30
Oct 30 16:50:40.872546 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.872468    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:50:40.956536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.956507    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1db025-db48-455f-8d67-0ee26e3f65f7-kube-api-access-g5nrb\" (UniqueName: \"kubernetes.io/projected/2f1db025-db48-455f-8d67-0ee26e3f65f7-kube-api-access-g5nrb\") pod \"2f1db025-db48-455f-8d67-0ee26e3f65f7\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") "
Oct 30 16:50:40.956731 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.956620    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1db025-db48-455f-8d67-0ee26e3f65f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1db025-db48-455f-8d67-0ee26e3f65f7-openshift-service-ca\") pod \"2f1db025-db48-455f-8d67-0ee26e3f65f7\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") "
Oct 30 16:50:40.956731 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.956646    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-metrics-certs\") pod \"2f1db025-db48-455f-8d67-0ee26e3f65f7\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") "
Oct 30 16:50:40.956731 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.956691    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-maas-api-tls\") pod \"2f1db025-db48-455f-8d67-0ee26e3f65f7\" (UID: \"2f1db025-db48-455f-8d67-0ee26e3f65f7\") "
Oct 30 16:50:40.957068 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.957040    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1db025-db48-455f-8d67-0ee26e3f65f7-openshift-service-ca" pod "2f1db025-db48-455f-8d67-0ee26e3f65f7" (UID: "2f1db025-db48-455f-8d67-0ee26e3f65f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:50:40.957354 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.957333    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1db025-db48-455f-8d67-0ee26e3f65f7-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:40.958773 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.958748    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1db025-db48-455f-8d67-0ee26e3f65f7-kube-api-access-g5nrb" pod "2f1db025-db48-455f-8d67-0ee26e3f65f7" (UID: "2f1db025-db48-455f-8d67-0ee26e3f65f7"). InnerVolumeSpecName "kube-api-access-g5nrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:40.959147 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.959129    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-metrics-certs" pod "2f1db025-db48-455f-8d67-0ee26e3f65f7" (UID: "2f1db025-db48-455f-8d67-0ee26e3f65f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:50:40.959311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:40.959290    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-maas-api-tls" pod "2f1db025-db48-455f-8d67-0ee26e3f65f7" (UID: "2f1db025-db48-455f-8d67-0ee26e3f65f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:50:41.061210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.060223    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:41.061210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.060261    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1db025-db48-455f-8d67-0ee26e3f65f7-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:41.061210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.060275    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5nrb\" (UniqueName: \"kubernetes.io/projected/2f1db025-db48-455f-8d67-0ee26e3f65f7-kube-api-access-g5nrb\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:41.134632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.134545    4497 generic.go:356] "Generic (PLEG): container finished" podID="2f1db025-db48-455f-8d67-0ee26e3f65f7" containerID="d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635" exitCode=0
Oct 30 16:50:41.135254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.134608    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"
Oct 30 16:50:41.135544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.134631    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl" event={"ID":"2f1db025-db48-455f-8d67-0ee26e3f65f7","Type":"ContainerDied","Data":"d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635"}
Oct 30 16:50:41.135647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.135587    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl" event={"ID":"2f1db025-db48-455f-8d67-0ee26e3f65f7","Type":"ContainerDied","Data":"0d6ba20e41324f38adff1542532a5c2d20dc6adbef18e3dbda989707a2c7a199"}
Oct 30 16:50:41.135647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.135633    4497 scope.go:122] "RemoveContainer" containerID="d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635"
Oct 30 16:50:41.149861 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.149844    4497 scope.go:122] "RemoveContainer" containerID="d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635"
Oct 30 16:50:41.150115 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:50:41.150099    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635\": container with ID starting with d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635 not found: ID does not exist" containerID="d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635"
Oct 30 16:50:41.150163 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.150122    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635"} err="failed to get container status \"d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635\": rpc error: code = NotFound desc = could not find container \"d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635\": container with ID starting with d811440488b519d8cbb47c2cbded6a982bead4125d8d872b70264393cc15c635 not found: ID does not exist"
Oct 30 16:50:41.183270 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.180945    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"]
Oct 30 16:50:41.184947 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:41.184917    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2e9431-7694f89bf9-7pvxl"]
Oct 30 16:50:42.299883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:42.299843    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1db025-db48-455f-8d67-0ee26e3f65f7" path="/var/lib/kubelet/pods/2f1db025-db48-455f-8d67-0ee26e3f65f7/volumes"
Oct 30 16:50:50.054023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:50.053988    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"]
Oct 30 16:50:50.054442 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:50.054259    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy" containerID="cri-o://1bc18da8ff8b3934d2555578ab3c4bfdbbd57ecdd0d79f990e9c6c3f14cc5404" gracePeriod=30
Oct 30 16:50:50.495684 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:50.495604    4497 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:50:50.495831 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:50.495678    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:50:51.007703 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:51.007670    4497 scope.go:122] "RemoveContainer" containerID="88367817e90f7f6ba6e7c443d1747a3819ae4a297c42ace13025269b2cbf3ade"
Oct 30 16:50:51.917552 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:50:51.917513    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:50:55.241387 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.241337    4497 generic.go:356] "Generic (PLEG): container finished" podID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerID="1bc18da8ff8b3934d2555578ab3c4bfdbbd57ecdd0d79f990e9c6c3f14cc5404" exitCode=0
Oct 30 16:50:55.241827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.241493    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" event={"ID":"55f80b1e-3475-477f-a6f3-d0069fedd05d","Type":"ContainerDied","Data":"1bc18da8ff8b3934d2555578ab3c4bfdbbd57ecdd0d79f990e9c6c3f14cc5404"}
Oct 30 16:50:55.318928 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.318862    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:50:55.491912 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.491879    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-kube-api-access-n8drh\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-kube-api-access-n8drh\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.491971    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f80b1e-3475-477f-a6f3-d0069fedd05d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f80b1e-3475-477f-a6f3-d0069fedd05d-istiod-ca-cert\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492090 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492073    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-envoy\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492156    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-podinfo\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492215    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-data\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492252    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-socket\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492294    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-credential-socket\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492365 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492332    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f80b1e-3475-477f-a6f3-d0069fedd05d-istiod-ca-cert" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:50:55.492476 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492454    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-certs\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492629    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-token\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-token\") pod \"55f80b1e-3475-477f-a6f3-d0069fedd05d\" (UID: \"55f80b1e-3475-477f-a6f3-d0069fedd05d\") "
Oct 30 16:50:55.492723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492537    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-socket" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:55.492723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492547    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-data" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:55.492831 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492722    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-credential-socket" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:55.492831 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.492735    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-certs" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:55.493391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.493367    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f80b1e-3475-477f-a6f3-d0069fedd05d-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:55.493477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.493395    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:55.493477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.493410    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:55.493477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.493423    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:55.493477 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.493437    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:55.494397 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.494373    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-podinfo" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:50:55.494706 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.494687    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-envoy" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:50:55.494777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.494714    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-kube-api-access-n8drh" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "kube-api-access-n8drh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:55.494840 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.494820    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-token" pod "55f80b1e-3475-477f-a6f3-d0069fedd05d" (UID: "55f80b1e-3475-477f-a6f3-d0069fedd05d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:50:55.595009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.594912    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8drh\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-kube-api-access-n8drh\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:55.595009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.594955    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:55.595009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.594965    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:55.595009 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:55.594973    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f80b1e-3475-477f-a6f3-d0069fedd05d-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:50:56.248567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:56.248532    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"
Oct 30 16:50:56.249035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:56.248534    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd" event={"ID":"55f80b1e-3475-477f-a6f3-d0069fedd05d","Type":"ContainerDied","Data":"237bf2d459a552f6a7967f8058946578b50632d03a8338a1a9749d1630f49b11"}
Oct 30 16:50:56.249035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:56.248678    4497 scope.go:122] "RemoveContainer" containerID="1bc18da8ff8b3934d2555578ab3c4bfdbbd57ecdd0d79f990e9c6c3f14cc5404"
Oct 30 16:50:56.304101 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:56.304072    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"]
Oct 30 16:50:56.313262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:56.313229    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-ng2rd"]
Oct 30 16:50:58.301859 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:50:58.301821    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" path="/var/lib/kubelet/pods/55f80b1e-3475-477f-a6f3-d0069fedd05d/volumes"
Oct 30 16:51:08.780737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:08.780700    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"]
Oct 30 16:51:08.781659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:08.781597    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d" podUID="824c439f-96b2-42d9-bed2-ff36ae080b2c" containerName="maas-api" containerID="cri-o://61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee" gracePeriod=30
Oct 30 16:51:09.054340 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.054276    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:51:09.213857 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.213826    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824c439f-96b2-42d9-bed2-ff36ae080b2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824c439f-96b2-42d9-bed2-ff36ae080b2c-openshift-service-ca\") pod \"824c439f-96b2-42d9-bed2-ff36ae080b2c\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") "
Oct 30 16:51:09.214059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.213923    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-maas-api-tls\") pod \"824c439f-96b2-42d9-bed2-ff36ae080b2c\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") "
Oct 30 16:51:09.214059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.213968    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824c439f-96b2-42d9-bed2-ff36ae080b2c-kube-api-access-dgtz5\" (UniqueName: \"kubernetes.io/projected/824c439f-96b2-42d9-bed2-ff36ae080b2c-kube-api-access-dgtz5\") pod \"824c439f-96b2-42d9-bed2-ff36ae080b2c\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") "
Oct 30 16:51:09.214059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.213992    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-metrics-certs\") pod \"824c439f-96b2-42d9-bed2-ff36ae080b2c\" (UID: \"824c439f-96b2-42d9-bed2-ff36ae080b2c\") "
Oct 30 16:51:09.214292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.214215    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824c439f-96b2-42d9-bed2-ff36ae080b2c-openshift-service-ca" pod "824c439f-96b2-42d9-bed2-ff36ae080b2c" (UID: "824c439f-96b2-42d9-bed2-ff36ae080b2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:51:09.214660 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.214635    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/824c439f-96b2-42d9-bed2-ff36ae080b2c-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:09.216232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.216175    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-maas-api-tls" pod "824c439f-96b2-42d9-bed2-ff36ae080b2c" (UID: "824c439f-96b2-42d9-bed2-ff36ae080b2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:51:09.216429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.216278    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-metrics-certs" pod "824c439f-96b2-42d9-bed2-ff36ae080b2c" (UID: "824c439f-96b2-42d9-bed2-ff36ae080b2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:51:09.216429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.216323    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824c439f-96b2-42d9-bed2-ff36ae080b2c-kube-api-access-dgtz5" pod "824c439f-96b2-42d9-bed2-ff36ae080b2c" (UID: "824c439f-96b2-42d9-bed2-ff36ae080b2c"). InnerVolumeSpecName "kube-api-access-dgtz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:51:09.315724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.315634    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:09.315724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.315669    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtz5\" (UniqueName: \"kubernetes.io/projected/824c439f-96b2-42d9-bed2-ff36ae080b2c-kube-api-access-dgtz5\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:09.315724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.315682    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/824c439f-96b2-42d9-bed2-ff36ae080b2c-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:09.336343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.336312    4497 generic.go:356] "Generic (PLEG): container finished" podID="824c439f-96b2-42d9-bed2-ff36ae080b2c" containerID="61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee" exitCode=0
Oct 30 16:51:09.336502 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.336378    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"
Oct 30 16:51:09.336502 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.336396    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d" event={"ID":"824c439f-96b2-42d9-bed2-ff36ae080b2c","Type":"ContainerDied","Data":"61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee"}
Oct 30 16:51:09.336502 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.336471    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d" event={"ID":"824c439f-96b2-42d9-bed2-ff36ae080b2c","Type":"ContainerDied","Data":"8274c52d6a52a6be2c10279552cc17d58ad3230a37169157ae53df7a9fdad3ce"}
Oct 30 16:51:09.336636 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.336503    4497 scope.go:122] "RemoveContainer" containerID="61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee"
Oct 30 16:51:09.346343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.346319    4497 scope.go:122] "RemoveContainer" containerID="61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee"
Oct 30 16:51:09.346673 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:09.346642    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee\": container with ID starting with 61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee not found: ID does not exist" containerID="61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee"
Oct 30 16:51:09.346807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.346681    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee"} err="failed to get container status \"61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee\": rpc error: code = NotFound desc = could not find container \"61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee\": container with ID starting with 61e9e4fc2d7ce7cdfdce6730a01eb98d06d5154608daffc1616e238a6582f9ee not found: ID does not exist"
Oct 30 16:51:09.398971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.391714    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"]
Oct 30 16:51:09.398971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:09.396330    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cfe9f0-68747d88cf-5vh5d"]
Oct 30 16:51:10.302825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:10.302781    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824c439f-96b2-42d9-bed2-ff36ae080b2c" path="/var/lib/kubelet/pods/824c439f-96b2-42d9-bed2-ff36ae080b2c/volumes"
Oct 30 16:51:17.988181 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:17.988139    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"]
Oct 30 16:51:17.988739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:17.988470    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy" containerID="cri-o://66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491" gracePeriod=30
Oct 30 16:51:18.676405 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:18.676366    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"]
Oct 30 16:51:18.676725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:18.676670    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy" containerID="cri-o://752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336" gracePeriod=30
Oct 30 16:51:19.824333 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:19.824297    4497 patch_prober.go:29] interesting pod/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:51:19.824715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:19.824364    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:51:20.715827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:20.715792    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"]
Oct 30 16:51:20.716043 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:20.716017    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy" containerID="cri-o://c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf" gracePeriod=30
Oct 30 16:51:21.916518 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:21.916476    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:51:23.322530 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.321943    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:51:23.398878 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.398835    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-envoy\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.399086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.398900    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-certs\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.399086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.398934    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-credential-socket\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.399086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.398969    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-kube-api-access-xl6mq\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-kube-api-access-xl6mq\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.399086 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.399056    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6964757-3358-4563-814a-89ba36f850da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6964757-3358-4563-814a-89ba36f850da-istio-podinfo\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.399370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.399164    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-data\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.399370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.399298    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-socket\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.399370 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.399332    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-istio-token\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-istio-token\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.399548 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.399426    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6964757-3358-4563-814a-89ba36f850da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6964757-3358-4563-814a-89ba36f850da-istiod-ca-cert\") pod \"d6964757-3358-4563-814a-89ba36f850da\" (UID: \"d6964757-3358-4563-814a-89ba36f850da\") "
Oct 30 16:51:23.400479 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.400308    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-socket" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:23.400747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.400704    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-certs" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:23.401464 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.401443    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-data" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:23.417336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.406618    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6964757-3358-4563-814a-89ba36f850da-istiod-ca-cert" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:51:23.417336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.407175    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-credential-socket" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:23.427858 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.427741    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-kube-api-access-xl6mq" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "kube-api-access-xl6mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:51:23.431012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.430109    4497 generic.go:356] "Generic (PLEG): container finished" podID="d6964757-3358-4563-814a-89ba36f850da" containerID="66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491" exitCode=0
Oct 30 16:51:23.431012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.430230    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" event={"ID":"d6964757-3358-4563-814a-89ba36f850da","Type":"ContainerDied","Data":"66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491"}
Oct 30 16:51:23.431012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.430287    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8" event={"ID":"d6964757-3358-4563-814a-89ba36f850da","Type":"ContainerDied","Data":"166cbaf14c725f747449bf3616c426b68bf219a31234564d9986568a620efc5d"}
Oct 30 16:51:23.431012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.430319    4497 scope.go:122] "RemoveContainer" containerID="66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491"
Oct 30 16:51:23.431012 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.430499    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"
Oct 30 16:51:23.437900 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.437846    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6964757-3358-4563-814a-89ba36f850da-istio-podinfo" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:51:23.440972 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.440933    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-envoy" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:23.441395 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.441096    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-istio-token" pod "d6964757-3358-4563-814a-89ba36f850da" (UID: "d6964757-3358-4563-814a-89ba36f850da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:51:23.448339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.448312    4497 scope.go:122] "RemoveContainer" containerID="e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2"
Oct 30 16:51:23.458541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.458519    4497 scope.go:122] "RemoveContainer" containerID="66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491"
Oct 30 16:51:23.458842 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:23.458821    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491\": container with ID starting with 66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491 not found: ID does not exist" containerID="66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491"
Oct 30 16:51:23.458934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.458856    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491"} err="failed to get container status \"66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491\": rpc error: code = NotFound desc = could not find container \"66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491\": container with ID starting with 66045b6d13fcf70dfa8af3ec907eef8380dc344b0aa3d41e1ba640bd78eb7491 not found: ID does not exist"
Oct 30 16:51:23.458934 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.458881    4497 scope.go:122] "RemoveContainer" containerID="e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2"
Oct 30 16:51:23.459215 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:23.459155    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2\": container with ID starting with e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2 not found: ID does not exist" containerID="e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2"
Oct 30 16:51:23.459261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.459227    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2"} err="failed to get container status \"e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2\": rpc error: code = NotFound desc = could not find container \"e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2\": container with ID starting with e1b24f521be168c9184dd64d311ada5826975a963387e2a8cc5a9fdd858e33e2 not found: ID does not exist"
Oct 30 16:51:23.501956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.501906    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6964757-3358-4563-814a-89ba36f850da-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.501956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.501951    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.501956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.501967    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.502309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.501980    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.502309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.501996    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6964757-3358-4563-814a-89ba36f850da-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.502309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.502009    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.502309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.502022    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.502309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.502036    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6964757-3358-4563-814a-89ba36f850da-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.502309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.502050    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6mq\" (UniqueName: \"kubernetes.io/projected/d6964757-3358-4563-814a-89ba36f850da-kube-api-access-xl6mq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:23.833213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.822208    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"]
Oct 30 16:51:23.833213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:23.826157    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c408f1-openshift-default-5f4584c4d9-ls5m8"]
Oct 30 16:51:24.131103 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.131073    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:51:24.259249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.259156    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-kube-api-access-b9wh8\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-kube-api-access-b9wh8\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.259297    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-podinfo\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.259345    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-credential-socket\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.259380    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-certs\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.259406    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-token\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-token\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.259456    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-socket\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.259573    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13fde00-4ff5-4167-84fc-5856dad1970c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13fde00-4ff5-4167-84fc-5856dad1970c-istiod-ca-cert\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.259810    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-credential-socket" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.260122    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.260852    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13fde00-4ff5-4167-84fc-5856dad1970c-istiod-ca-cert" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.261015    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-socket" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.261203    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-certs" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:24.263690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.262942    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-kube-api-access-b9wh8" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "kube-api-access-b9wh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:51:24.264906 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.264869    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-token" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:51:24.265423 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.265392    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-podinfo" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:51:24.306925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.306708    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6964757-3358-4563-814a-89ba36f850da" path="/var/lib/kubelet/pods/d6964757-3358-4563-814a-89ba36f850da/volumes"
Oct 30 16:51:24.360705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.360671    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-data\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.360705 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.360713    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-envoy\") pod \"c13fde00-4ff5-4167-84fc-5856dad1970c\" (UID: \"c13fde00-4ff5-4167-84fc-5856dad1970c\") "
Oct 30 16:51:24.361241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.360919    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-data" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:24.361241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.361103    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.361241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.361116    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13fde00-4ff5-4167-84fc-5856dad1970c-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.361241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.361125    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.361241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.361135    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9wh8\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-kube-api-access-b9wh8\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.361241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.361143    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.361241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.361152    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.361241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.361160    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.362763 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.362742    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-envoy" pod "c13fde00-4ff5-4167-84fc-5856dad1970c" (UID: "c13fde00-4ff5-4167-84fc-5856dad1970c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:24.439908 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.439874    4497 generic.go:356] "Generic (PLEG): container finished" podID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerID="752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336" exitCode=0
Oct 30 16:51:24.440088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.439942    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"
Oct 30 16:51:24.440088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.439954    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" event={"ID":"c13fde00-4ff5-4167-84fc-5856dad1970c","Type":"ContainerDied","Data":"752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336"}
Oct 30 16:51:24.440088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.440008    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t" event={"ID":"c13fde00-4ff5-4167-84fc-5856dad1970c","Type":"ContainerDied","Data":"47604144eb1128adde5a0043028e60580dc2557fe237f007a313799c963c0f01"}
Oct 30 16:51:24.440088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.440030    4497 scope.go:122] "RemoveContainer" containerID="752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336"
Oct 30 16:51:24.449826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.449805    4497 scope.go:122] "RemoveContainer" containerID="a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b"
Oct 30 16:51:24.458921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.458900    4497 scope.go:122] "RemoveContainer" containerID="752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336"
Oct 30 16:51:24.459255 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:24.459233    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336\": container with ID starting with 752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336 not found: ID does not exist" containerID="752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336"
Oct 30 16:51:24.459343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.459270    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336"} err="failed to get container status \"752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336\": rpc error: code = NotFound desc = could not find container \"752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336\": container with ID starting with 752bc7edfa299c1dfb40ca37ed1d5b84389af9a8afe6c70679e1bcd68c842336 not found: ID does not exist"
Oct 30 16:51:24.459343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.459294    4497 scope.go:122] "RemoveContainer" containerID="a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b"
Oct 30 16:51:24.459540 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:24.459522    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b\": container with ID starting with a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b not found: ID does not exist" containerID="a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b"
Oct 30 16:51:24.459583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.459547    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b"} err="failed to get container status \"a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b\": rpc error: code = NotFound desc = could not find container \"a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b\": container with ID starting with a2b66fef9bcee72c123c712b9234c4a68194c9566d38dc5d5063f055f197970b not found: ID does not exist"
Oct 30 16:51:24.462310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.462293    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13fde00-4ff5-4167-84fc-5856dad1970c-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:24.488182 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.488144    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"]
Oct 30 16:51:24.498430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:24.498404    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-cfe9f0-openshift-default-8465d4b54f-22f7t"]
Oct 30 16:51:25.980759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:25.980695    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:51:26.079697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.079665    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-token\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-token\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.079896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.079732    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-certs\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.079896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.079772    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f5c38f-62d7-4096-8f06-c90062d15a82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f5c38f-62d7-4096-8f06-c90062d15a82-istiod-ca-cert\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.079896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.079791    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-data\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.079896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.079833    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-socket\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.079896 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.079887    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-kube-api-access-hfr4k\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-kube-api-access-hfr4k\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.080159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.079914    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-credential-socket\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.080159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.079970    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-podinfo\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.080159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080023    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-envoy\") pod \"f5f5c38f-62d7-4096-8f06-c90062d15a82\" (UID: \"f5f5c38f-62d7-4096-8f06-c90062d15a82\") "
Oct 30 16:51:26.080343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080142    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-data" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:26.080343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080211    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-socket" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:26.080343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080262    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f5c38f-62d7-4096-8f06-c90062d15a82-istiod-ca-cert" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:51:26.080503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080399    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-credential-socket" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:26.080557 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080515    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-certs" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:26.080673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080655    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.080743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080673    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5f5c38f-62d7-4096-8f06-c90062d15a82-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.080743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080682    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.080743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080690    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.080743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.080700    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.082164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.082136    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-token" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:51:26.082164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.082142    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-podinfo" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:51:26.082358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.082173    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-envoy" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:51:26.082358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.082321    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-kube-api-access-hfr4k" pod "f5f5c38f-62d7-4096-8f06-c90062d15a82" (UID: "f5f5c38f-62d7-4096-8f06-c90062d15a82"). InnerVolumeSpecName "kube-api-access-hfr4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:51:26.181625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.181593    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfr4k\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-kube-api-access-hfr4k\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.181625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.181621    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.181625 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.181632    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.181864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.181639    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5f5c38f-62d7-4096-8f06-c90062d15a82-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:26.299992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.299947    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" path="/var/lib/kubelet/pods/c13fde00-4ff5-4167-84fc-5856dad1970c/volumes"
Oct 30 16:51:26.454074 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.454042    4497 generic.go:356] "Generic (PLEG): container finished" podID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerID="c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf" exitCode=0
Oct 30 16:51:26.454276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.454099    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"
Oct 30 16:51:26.454276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.454133    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" event={"ID":"f5f5c38f-62d7-4096-8f06-c90062d15a82","Type":"ContainerDied","Data":"c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf"}
Oct 30 16:51:26.454276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.454223    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl" event={"ID":"f5f5c38f-62d7-4096-8f06-c90062d15a82","Type":"ContainerDied","Data":"0dec42eed6e705d03187fa57a8c0e2ae564b10274bb8d3901a9aa23ba67cb99f"}
Oct 30 16:51:26.454276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.454253    4497 scope.go:122] "RemoveContainer" containerID="c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf"
Oct 30 16:51:26.463774 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.463754    4497 scope.go:122] "RemoveContainer" containerID="f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae"
Oct 30 16:51:26.472936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.472913    4497 scope.go:122] "RemoveContainer" containerID="c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf"
Oct 30 16:51:26.473250 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:26.473229    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf\": container with ID starting with c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf not found: ID does not exist" containerID="c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf"
Oct 30 16:51:26.473302 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.473261    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf"} err="failed to get container status \"c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf\": rpc error: code = NotFound desc = could not find container \"c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf\": container with ID starting with c296c4dbd06559aaf6e8f6ca8f4fb3193b1dc3da110dcd9159cdc85db03803bf not found: ID does not exist"
Oct 30 16:51:26.473302 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.473279    4497 scope.go:122] "RemoveContainer" containerID="f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae"
Oct 30 16:51:26.473538 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:26.473517    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae\": container with ID starting with f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae not found: ID does not exist" containerID="f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae"
Oct 30 16:51:26.473582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.473547    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae"} err="failed to get container status \"f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae\": rpc error: code = NotFound desc = could not find container \"f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae\": container with ID starting with f917036fa7c3fc03c3f90b182f3c1879ea55f3a9766b8dfab8eaa251bef7b5ae not found: ID does not exist"
Oct 30 16:51:26.504071 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.504040    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"]
Oct 30 16:51:26.514034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:26.514008    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-06ac18-openshift-default-5b8b5b5679-6q8rl"]
Oct 30 16:51:28.300288 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:28.300252    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" path="/var/lib/kubelet/pods/f5f5c38f-62d7-4096-8f06-c90062d15a82/volumes"
Oct 30 16:51:29.809747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.809709    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"]
Oct 30 16:51:29.810394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810363    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy"
Oct 30 16:51:29.810394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810397    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810412    4497 cpu_manager.go:414] "containerMap: removing container" podUID="824c439f-96b2-42d9-bed2-ff36ae080b2c" containerName="maas-api"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810419    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824c439f-96b2-42d9-bed2-ff36ae080b2c" containerName="maas-api"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810428    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810434    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810445    4497 cpu_manager.go:414] "containerMap: removing container" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810451    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810458    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ce395c5c-0da1-4972-bb6a-0fba209464a9" containerName="maas-api"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810462    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce395c5c-0da1-4972-bb6a-0fba209464a9" containerName="maas-api"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810474    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810478    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810486    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810491    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy"
Oct 30 16:51:29.810503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810508    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2f1db025-db48-455f-8d67-0ee26e3f65f7" containerName="maas-api"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810515    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1db025-db48-455f-8d67-0ee26e3f65f7" containerName="maas-api"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810529    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810533    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810543    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a8a6167e-9755-42ad-9d58-becd7a81cf2c" containerName="maas-api"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810547    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a6167e-9755-42ad-9d58-becd7a81cf2c" containerName="maas-api"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810555    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" containerName="discovery"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810559    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" containerName="discovery"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810573    4497 cpu_manager.go:414] "containerMap: removing container" podUID="95a82982-3484-4b87-b94c-2217a310ffa8" containerName="payload-pre-processing"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810577    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a82982-3484-4b87-b94c-2217a310ffa8" containerName="payload-pre-processing"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810583    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3179bbae-5e62-48e2-a016-9f9da83dcb39" containerName="payload-processing"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810587    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3179bbae-5e62-48e2-a016-9f9da83dcb39" containerName="payload-processing"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810596    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810603    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810684    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a6167e-9755-42ad-9d58-becd7a81cf2c" containerName="maas-api"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810696    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef2cc7e-99c1-4d82-bfdf-242fbd0440aa" containerName="discovery"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810702    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a82982-3484-4b87-b94c-2217a310ffa8" containerName="payload-pre-processing"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810707    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3179bbae-5e62-48e2-a016-9f9da83dcb39" containerName="payload-processing"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810712    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810717    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810723    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810728    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810733    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="824c439f-96b2-42d9-bed2-ff36ae080b2c" containerName="maas-api"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810738    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810744    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f80b1e-3475-477f-a6f3-d0069fedd05d" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810751    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce395c5c-0da1-4972-bb6a-0fba209464a9" containerName="maas-api"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810761    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810769    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6964757-3358-4563-814a-89ba36f850da" containerName="istio-proxy"
Oct 30 16:51:29.810925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.810777    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1db025-db48-455f-8d67-0ee26e3f65f7" containerName="maas-api"
Oct 30 16:51:29.815691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.815671    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:29.827764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.827736    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-52f3fd\""
Oct 30 16:51:29.841266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.837596    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"]
Oct 30 16:51:29.874015 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.873977    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"]
Oct 30 16:51:29.874854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.874829    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy"
Oct 30 16:51:29.874854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.874854    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b213dc75-c4b2-4ff1-8735-5d4d614b32d2" containerName="istio-proxy"
Oct 30 16:51:29.875010 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.874881    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy"
Oct 30 16:51:29.875010 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.874890    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13fde00-4ff5-4167-84fc-5856dad1970c" containerName="istio-proxy"
Oct 30 16:51:29.875134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.875024    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f5c38f-62d7-4096-8f06-c90062d15a82" containerName="istio-proxy"
Oct 30 16:51:29.879010 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.878980    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:29.896503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.896469    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"]
Oct 30 16:51:29.897461 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.897438    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-52f3fd-dockercfg-x75vg\""
Oct 30 16:51:29.923840 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.923802    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38fe36-c59b-4a43-97f3-3a55140a28e1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h\" (UID: \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:29.924035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.923851    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drp9h\" (UniqueName: \"kubernetes.io/projected/ce38fe36-c59b-4a43-97f3-3a55140a28e1-kube-api-access-drp9h\") pod \"payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h\" (UID: \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:29.924035 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.923989    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9361fbab-d01a-426c-9669-589c357bb916-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh\" (UID: \"9361fbab-d01a-426c-9669-589c357bb916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:29.924198 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.924165    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kx49\" (UniqueName: \"kubernetes.io/projected/9361fbab-d01a-426c-9669-589c357bb916-kube-api-access-8kx49\") pod \"payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh\" (UID: \"9361fbab-d01a-426c-9669-589c357bb916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:29.941904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.941864    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"]
Oct 30 16:51:29.946064 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.946032    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:29.970470 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:29.970413    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"]
Oct 30 16:51:30.025261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025219    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025302    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee2e057-5724-4f65-bfe4-46cf91386696-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025337    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9361fbab-d01a-426c-9669-589c357bb916-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh\" (UID: \"9361fbab-d01a-426c-9669-589c357bb916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:30.025430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025377    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025406    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025450    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025511    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kx49\" (UniqueName: \"kubernetes.io/projected/9361fbab-d01a-426c-9669-589c357bb916-kube-api-access-8kx49\") pod \"payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh\" (UID: \"9361fbab-d01a-426c-9669-589c357bb916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:30.025633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025568    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025593    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025625    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwfq\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-kube-api-access-9cwfq\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025656    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38fe36-c59b-4a43-97f3-3a55140a28e1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h\" (UID: \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:30.025730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025671    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.025730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.025689    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drp9h\" (UniqueName: \"kubernetes.io/projected/ce38fe36-c59b-4a43-97f3-3a55140a28e1-kube-api-access-drp9h\") pod \"payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h\" (UID: \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:30.026931 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.026903    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9361fbab-d01a-426c-9669-589c357bb916-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh\" (UID: \"9361fbab-d01a-426c-9669-589c357bb916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:30.027055 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.026903    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38fe36-c59b-4a43-97f3-3a55140a28e1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h\" (UID: \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:30.047050 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.047004    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drp9h\" (UniqueName: \"kubernetes.io/projected/ce38fe36-c59b-4a43-97f3-3a55140a28e1-kube-api-access-drp9h\") pod \"payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h\" (UID: \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:30.047226 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.047154    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kx49\" (UniqueName: \"kubernetes.io/projected/9361fbab-d01a-426c-9669-589c357bb916-kube-api-access-8kx49\") pod \"payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh\" (UID: \"9361fbab-d01a-426c-9669-589c357bb916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:30.126719 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.126615    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.126900 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.126749    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.126900 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.126784    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.126900 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.126871    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwfq\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-kube-api-access-9cwfq\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127061 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127000    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127057    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127096    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee2e057-5724-4f65-bfe4-46cf91386696-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127152    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127203    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127280    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127453 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127342    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127453 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127361    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.127684 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.127661    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.128104 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.128021    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee2e057-5724-4f65-bfe4-46cf91386696-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.129349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.129327    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:30.129631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.129609    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.129992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.129962    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.147763 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.147733    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.147876 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.147830    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwfq\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-kube-api-access-9cwfq\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.198491 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.198452    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:30.299828 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.299778    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:30.439304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.439276    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"]
Oct 30 16:51:30.443218 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:51:30.443170    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce38fe36_c59b_4a43_97f3_3a55140a28e1.slice/crio-0cdc277a9f3e4adc2cbba330344fcda3b4f7928847545b7d5194757346a8030c WatchSource:0}: Error finding container 0cdc277a9f3e4adc2cbba330344fcda3b4f7928847545b7d5194757346a8030c: Status 404 returned error can't find the container with id 0cdc277a9f3e4adc2cbba330344fcda3b4f7928847545b7d5194757346a8030c
Oct 30 16:51:30.481977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.481945    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h" event={"ID":"ce38fe36-c59b-4a43-97f3-3a55140a28e1","Type":"ContainerStarted","Data":"0cdc277a9f3e4adc2cbba330344fcda3b4f7928847545b7d5194757346a8030c"}
Oct 30 16:51:30.517550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.517523    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"]
Oct 30 16:51:30.521125 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:51:30.521100    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9361fbab_d01a_426c_9669_589c357bb916.slice/crio-6e5748ce237b91a48e20d6f884bb889d0dcd5d0cc914346c2c8b9e9d6bd4c7ff WatchSource:0}: Error finding container 6e5748ce237b91a48e20d6f884bb889d0dcd5d0cc914346c2c8b9e9d6bd4c7ff: Status 404 returned error can't find the container with id 6e5748ce237b91a48e20d6f884bb889d0dcd5d0cc914346c2c8b9e9d6bd4c7ff
Oct 30 16:51:30.616678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.616647    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"]
Oct 30 16:51:30.620710 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:51:30.620678    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee2e057_5724_4f65_bfe4_46cf91386696.slice/crio-60c3e85e1ce15f34a0878672b05f49ea15552be8ce1c1465e2bf988160287a85 WatchSource:0}: Error finding container 60c3e85e1ce15f34a0878672b05f49ea15552be8ce1c1465e2bf988160287a85: Status 404 returned error can't find the container with id 60c3e85e1ce15f34a0878672b05f49ea15552be8ce1c1465e2bf988160287a85
Oct 30 16:51:30.623674 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.623634    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:30.623775 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.623730    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:30.623816 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:30.623772    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:31.502623 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.502584    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" event={"ID":"7ee2e057-5724-4f65-bfe4-46cf91386696","Type":"ContainerStarted","Data":"f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4"}
Oct 30 16:51:31.503165 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.502650    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" event={"ID":"7ee2e057-5724-4f65-bfe4-46cf91386696","Type":"ContainerStarted","Data":"60c3e85e1ce15f34a0878672b05f49ea15552be8ce1c1465e2bf988160287a85"}
Oct 30 16:51:31.504227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.504203    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h" event={"ID":"ce38fe36-c59b-4a43-97f3-3a55140a28e1","Type":"ContainerStarted","Data":"34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18"}
Oct 30 16:51:31.504335 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.504251    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:31.506224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.506180    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh" event={"ID":"9361fbab-d01a-426c-9669-589c357bb916","Type":"ContainerStarted","Data":"5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425"}
Oct 30 16:51:31.506310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.506234    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh" event={"ID":"9361fbab-d01a-426c-9669-589c357bb916","Type":"ContainerStarted","Data":"6e5748ce237b91a48e20d6f884bb889d0dcd5d0cc914346c2c8b9e9d6bd4c7ff"}
Oct 30 16:51:31.506310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.506262    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:31.563755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.563699    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" podStartSLOduration=2.563685541 podStartE2EDuration="2.563685541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:31.553176975 +0000 UTC m=+1721.407055240" watchObservedRunningTime="2026-10-30 16:51:31.563685541 +0000 UTC m=+1721.417563806"
Oct 30 16:51:31.599261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.599176    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh" podStartSLOduration=2.599154219 podStartE2EDuration="2.599154219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:31.593462692 +0000 UTC m=+1721.447340958" watchObservedRunningTime="2026-10-30 16:51:31.599154219 +0000 UTC m=+1721.453032485"
Oct 30 16:51:31.630174 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:31.630121    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h" podStartSLOduration=2.630106596 podStartE2EDuration="2.630106596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:31.628927528 +0000 UTC m=+1721.482805792" watchObservedRunningTime="2026-10-30 16:51:31.630106596 +0000 UTC m=+1721.483984862"
Oct 30 16:51:32.304882 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:32.304848    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 16:51:32.305034 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:32.304903    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 16:51:32.306694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:32.306674    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:33.300805 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:33.300756    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 16:51:33.301316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:33.300823    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 16:51:34.300467 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:34.300431    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 16:51:34.300650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:34.300493    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 16:51:35.300753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:35.300717    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 16:51:35.301127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:35.300788    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 16:51:36.301004 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:36.300970    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 16:51:36.301436 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:36.301023    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 16:51:37.300975 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:37.300935    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 16:51:37.301164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:37.301019    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 16:51:38.305663 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:38.305627    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:38.306169 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:38.306147    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:38.307126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:38.307104    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 16:51:41.940521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:41.940486    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"]
Oct 30 16:51:41.946498 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:41.946470    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:41.959668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:41.959645    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-52f3fd-serving-cert\""
Oct 30 16:51:41.972031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:41.972005    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"]
Oct 30 16:51:42.097181 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.097148    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-metrics-certs\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.097405 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.097216    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d2a86d-4ccb-453f-a07a-48af87c66e38-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.097405 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.097312    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxzs\" (UniqueName: \"kubernetes.io/projected/a0d2a86d-4ccb-453f-a07a-48af87c66e38-kube-api-access-fnxzs\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.097504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.097421    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-maas-api-tls\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.205560 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.205469    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-maas-api-tls\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.205726 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.205695    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-metrics-certs\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.205771 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.205751    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d2a86d-4ccb-453f-a07a-48af87c66e38-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.205832 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.205808    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxzs\" (UniqueName: \"kubernetes.io/projected/a0d2a86d-4ccb-453f-a07a-48af87c66e38-kube-api-access-fnxzs\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.206766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.206735    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d2a86d-4ccb-453f-a07a-48af87c66e38-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.208984 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.208955    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-metrics-certs\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.217655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.217627    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-maas-api-tls\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.227334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.227309    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxzs\" (UniqueName: \"kubernetes.io/projected/a0d2a86d-4ccb-453f-a07a-48af87c66e38-kube-api-access-fnxzs\") pod \"maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.263933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.263902    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:42.515358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.515333    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:51:42.515536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.515410    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:51:42.567854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.567803    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"]
Oct 30 16:51:42.577232 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:51:42.577205    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d2a86d_4ccb_453f_a07a_48af87c66e38.slice/crio-26d1c79420b00808ae4fe1848d1df4f3d7224442a644d89f0f0a0513a7070e69 WatchSource:0}: Error finding container 26d1c79420b00808ae4fe1848d1df4f3d7224442a644d89f0f0a0513a7070e69: Status 404 returned error can't find the container with id 26d1c79420b00808ae4fe1848d1df4f3d7224442a644d89f0f0a0513a7070e69
Oct 30 16:51:42.578836 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:42.578816    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:51:43.580469 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:43.580436    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8" event={"ID":"a0d2a86d-4ccb-453f-a07a-48af87c66e38","Type":"ContainerStarted","Data":"f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c"}
Oct 30 16:51:43.580942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:43.580489    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8" event={"ID":"a0d2a86d-4ccb-453f-a07a-48af87c66e38","Type":"ContainerStarted","Data":"26d1c79420b00808ae4fe1848d1df4f3d7224442a644d89f0f0a0513a7070e69"}
Oct 30 16:51:43.580942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:43.580511    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:43.623273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:43.622948    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8" podStartSLOduration=2.179257383 podStartE2EDuration="2.622930867s" totalImagesPullingTime="443.673484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:43.620397992 +0000 UTC m=+1733.474276258" watchObservedRunningTime="2026-10-30 16:51:43.622930867 +0000 UTC m=+1733.476809133"
Oct 30 16:51:45.004211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.004096    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"]
Oct 30 16:51:45.032210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.027411    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"]
Oct 30 16:51:45.032210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.027539    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.047850    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.047917    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbjtr\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-kube-api-access-bbjtr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.047954    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.048019    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.048052    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.048094    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.048146    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b81f7d-4567-46d4-a0b7-1647e183f814-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.048210    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.050381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.048268    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.151764    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.151843    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.151937    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.151989    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbjtr\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-kube-api-access-bbjtr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.152024    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.152093    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.152126    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.152169    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.152825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.152243    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b81f7d-4567-46d4-a0b7-1647e183f814-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.173237 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.153056    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b81f7d-4567-46d4-a0b7-1647e183f814-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.202210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.201937    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.202381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.202240    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.237211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.236520    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.318257 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.314662    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.318257 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.314957    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.393213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.392623    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.424212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.415493    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.471210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.461947    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbjtr\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-kube-api-access-bbjtr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.671025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.670943    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:45.797889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.767350    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"]
Oct 30 16:51:45.797889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:45.767692    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7" podUID="bbb85f85-4cab-4b18-bff7-bc7bb13967aa" containerName="maas-api" containerID="cri-o://0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be" gracePeriod=30
Oct 30 16:51:46.128360 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.128330    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:51:46.167431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.167400    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-maas-api-tls\") pod \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") "
Oct 30 16:51:46.167619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.167529    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-metrics-certs\") pod \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") "
Oct 30 16:51:46.167619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.167573    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-kube-api-access-z7xdd\" (UniqueName: \"kubernetes.io/projected/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-kube-api-access-z7xdd\") pod \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") "
Oct 30 16:51:46.167619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.167605    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-openshift-service-ca\") pod \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\" (UID: \"bbb85f85-4cab-4b18-bff7-bc7bb13967aa\") "
Oct 30 16:51:46.168257 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.168230    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-openshift-service-ca" pod "bbb85f85-4cab-4b18-bff7-bc7bb13967aa" (UID: "bbb85f85-4cab-4b18-bff7-bc7bb13967aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:51:46.170170 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.170136    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-metrics-certs" pod "bbb85f85-4cab-4b18-bff7-bc7bb13967aa" (UID: "bbb85f85-4cab-4b18-bff7-bc7bb13967aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:51:46.170689 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.170639    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-maas-api-tls" pod "bbb85f85-4cab-4b18-bff7-bc7bb13967aa" (UID: "bbb85f85-4cab-4b18-bff7-bc7bb13967aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:51:46.170801 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.170767    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-kube-api-access-z7xdd" pod "bbb85f85-4cab-4b18-bff7-bc7bb13967aa" (UID: "bbb85f85-4cab-4b18-bff7-bc7bb13967aa"). InnerVolumeSpecName "kube-api-access-z7xdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:51:46.219629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.219509    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"]
Oct 30 16:51:46.223665 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:51:46.223635    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b81f7d_4567_46d4_a0b7_1647e183f814.slice/crio-4edf8acd1e6da5dcc99bd31e36a75a8ba42259bec4e47b8b826e50fcc1b40211 WatchSource:0}: Error finding container 4edf8acd1e6da5dcc99bd31e36a75a8ba42259bec4e47b8b826e50fcc1b40211: Status 404 returned error can't find the container with id 4edf8acd1e6da5dcc99bd31e36a75a8ba42259bec4e47b8b826e50fcc1b40211
Oct 30 16:51:46.226403 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.226366    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:46.226478 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.226435    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:46.226478 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.226467    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:46.269627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.269600    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xdd\" (UniqueName: \"kubernetes.io/projected/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-kube-api-access-z7xdd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:46.269627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.269628    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:46.269789 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.269638    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:46.269789 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.269648    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbb85f85-4cab-4b18-bff7-bc7bb13967aa-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:51:46.624719 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.624684    4497 generic.go:356] "Generic (PLEG): container finished" podID="bbb85f85-4cab-4b18-bff7-bc7bb13967aa" containerID="0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be" exitCode=0
Oct 30 16:51:46.624904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.624746    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"
Oct 30 16:51:46.624904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.624768    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7" event={"ID":"bbb85f85-4cab-4b18-bff7-bc7bb13967aa","Type":"ContainerDied","Data":"0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be"}
Oct 30 16:51:46.624904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.624825    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7" event={"ID":"bbb85f85-4cab-4b18-bff7-bc7bb13967aa","Type":"ContainerDied","Data":"5b6d7f7e38fd39e26d87118db119600f7cd823121ed2d473563952478f466419"}
Oct 30 16:51:46.624904 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.624851    4497 scope.go:122] "RemoveContainer" containerID="0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be"
Oct 30 16:51:46.626867 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.626841    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" event={"ID":"d0b81f7d-4567-46d4-a0b7-1647e183f814","Type":"ContainerStarted","Data":"d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292"}
Oct 30 16:51:46.627016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.626889    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" event={"ID":"d0b81f7d-4567-46d4-a0b7-1647e183f814","Type":"ContainerStarted","Data":"4edf8acd1e6da5dcc99bd31e36a75a8ba42259bec4e47b8b826e50fcc1b40211"}
Oct 30 16:51:46.635084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.635057    4497 scope.go:122] "RemoveContainer" containerID="0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be"
Oct 30 16:51:46.635402 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:46.635376    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be\": container with ID starting with 0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be not found: ID does not exist" containerID="0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be"
Oct 30 16:51:46.635501 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.635412    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be"} err="failed to get container status \"0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be\": rpc error: code = NotFound desc = could not find container \"0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be\": container with ID starting with 0d2d757c89e28eccc2292dd416554034f80c99e46189f779d94f4086060181be not found: ID does not exist"
Oct 30 16:51:46.671797 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.671769    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:46.687351 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.687301    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podStartSLOduration=2.687283843 podStartE2EDuration="2.687283843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:46.676306239 +0000 UTC m=+1736.530184503" watchObservedRunningTime="2026-10-30 16:51:46.687283843 +0000 UTC m=+1736.541162109"
Oct 30 16:51:46.708079 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.708044    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"]
Oct 30 16:51:46.711873 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:46.711834    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d5903-86ff745d8-qswq7"]
Oct 30 16:51:47.681778 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:47.681730    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 16:51:47.682377 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:47.682346    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 16:51:48.302714 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:48.302678    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb85f85-4cab-4b18-bff7-bc7bb13967aa" path="/var/lib/kubelet/pods/bbb85f85-4cab-4b18-bff7-bc7bb13967aa/volumes"
Oct 30 16:51:48.696269 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:48.681057    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 16:51:48.696775 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:48.696290    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 16:51:49.601574 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:49.601539    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:51:49.675584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:49.675547    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 16:51:49.675843 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:49.675817    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 16:51:50.690211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:50.678591    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 16:51:50.690211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:50.678652    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 16:51:51.691211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:51.677575    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 16:51:51.691211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:51.677639    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 16:51:51.924265 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:51:51.924227    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:51:52.672100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:52.672058    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 16:51:52.672314 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:52.672122    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 16:51:53.808014 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:53.807967    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:53.808549 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:53.808339    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:53.809657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:53.809634    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 16:51:56.653965 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.653928    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"]
Oct 30 16:51:56.654803 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.654781    4497 cpu_manager.go:414] "containerMap: removing container" podUID="bbb85f85-4cab-4b18-bff7-bc7bb13967aa" containerName="maas-api"
Oct 30 16:51:56.654855 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.654807    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb85f85-4cab-4b18-bff7-bc7bb13967aa" containerName="maas-api"
Oct 30 16:51:56.655019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.655006    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb85f85-4cab-4b18-bff7-bc7bb13967aa" containerName="maas-api"
Oct 30 16:51:56.661974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.661949    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.674346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.674289    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-07cb3c-openshift-default-dockercfg-9vsff\""
Oct 30 16:51:56.686988 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.686956    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"]
Oct 30 16:51:56.704992 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.704962    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-socket\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.705165 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.705009    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-istio-token\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.705165 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.705120    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-data\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.705262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.705197    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbq4\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-kube-api-access-xlbq4\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.705347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.705331    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-credential-socket\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.705385 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.705357    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-certs\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.705431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.705392    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-envoy\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.705525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.705503    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42284a65-b5fb-4b79-b717-1961164e79ce-istio-podinfo\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.705591 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.705542    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42284a65-b5fb-4b79-b717-1961164e79ce-istiod-ca-cert\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807435    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-istio-token\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807526    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-data\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807588    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbq4\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-kube-api-access-xlbq4\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807710    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-credential-socket\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807741    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-certs\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807782    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-envoy\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807821    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42284a65-b5fb-4b79-b717-1961164e79ce-istio-podinfo\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807849    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42284a65-b5fb-4b79-b717-1961164e79ce-istiod-ca-cert\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.808327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.807895    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-socket\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.809726 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.809561    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-socket\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.810646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.810625    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-data\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.811457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.811422    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-credential-socket\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.811889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.811862    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-certs\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.820881 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.820847    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42284a65-b5fb-4b79-b717-1961164e79ce-istiod-ca-cert\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.824484 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.824455    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-envoy\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.824847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.824823    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42284a65-b5fb-4b79-b717-1961164e79ce-istio-podinfo\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.832617 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.832592    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-istio-token\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.832979 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.832953    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbq4\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-kube-api-access-xlbq4\") pod \"e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") " pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:56.977572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:56.977491    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:57.310702 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:57.310666    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"]
Oct 30 16:51:57.314363 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:51:57.314338    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42284a65_b5fb_4b79_b717_1961164e79ce.slice/crio-a8f4a83d7e6b269e15df928b193a229e22f8d773d3883852d7ee89dc06f3b754 WatchSource:0}: Error finding container a8f4a83d7e6b269e15df928b193a229e22f8d773d3883852d7ee89dc06f3b754: Status 404 returned error can't find the container with id a8f4a83d7e6b269e15df928b193a229e22f8d773d3883852d7ee89dc06f3b754
Oct 30 16:51:57.316777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:57.316743    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:57.316854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:57.316811    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:57.316854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:57.316838    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:51:57.700963 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:57.700867    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" event={"ID":"42284a65-b5fb-4b79-b717-1961164e79ce","Type":"ContainerStarted","Data":"6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90"}
Oct 30 16:51:57.701461 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:57.701430    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" event={"ID":"42284a65-b5fb-4b79-b717-1961164e79ce","Type":"ContainerStarted","Data":"a8f4a83d7e6b269e15df928b193a229e22f8d773d3883852d7ee89dc06f3b754"}
Oct 30 16:51:57.743714 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:57.743653    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" podStartSLOduration=1.743635145 podStartE2EDuration="1.743635145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:57.74086921 +0000 UTC m=+1747.594747475" watchObservedRunningTime="2026-10-30 16:51:57.743635145 +0000 UTC m=+1747.597513411"
Oct 30 16:51:57.978089 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:57.977995    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:58.983596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:58.983565    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:59.712778 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:59.712746    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:51:59.713671 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:51:59.713653    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:52:01.123059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.123025    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"]
Oct 30 16:52:01.127417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.127395    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.164937 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.164904    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"]
Oct 30 16:52:01.266821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.266774    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.266984 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.266845    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.266984 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.266953    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwz4\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-kube-api-access-mpwz4\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.267125 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.267061    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22350f1f-626e-41f4-be41-aa75b103daea-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.267208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.267164    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.267291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.267275    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.267351 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.267340    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.267442 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.267428    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22350f1f-626e-41f4-be41-aa75b103daea-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.267483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.267453    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369029    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwz4\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-kube-api-access-mpwz4\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369280 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369130    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22350f1f-626e-41f4-be41-aa75b103daea-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369280 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369257    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369406 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369296    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369406 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369382    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369446    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22350f1f-626e-41f4-be41-aa75b103daea-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369485    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369567    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369662    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.369960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.369929    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.370088 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.370063    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.370164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.370128    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.370164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.370153    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.370350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.370328    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22350f1f-626e-41f4-be41-aa75b103daea-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.371828 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.371802    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.371911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.371865    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22350f1f-626e-41f4-be41-aa75b103daea-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.389891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.389827    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.390017 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.389947    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwz4\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-kube-api-access-mpwz4\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.441668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.441632    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:01.778274 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.778237    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"]
Oct 30 16:52:01.781509 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:01.781478    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22350f1f_626e_41f4_be41_aa75b103daea.slice/crio-52e9a202ef48d259923023396365b83f50df06cd631e55373430f6574c9b5cc1 WatchSource:0}: Error finding container 52e9a202ef48d259923023396365b83f50df06cd631e55373430f6574c9b5cc1: Status 404 returned error can't find the container with id 52e9a202ef48d259923023396365b83f50df06cd631e55373430f6574c9b5cc1
Oct 30 16:52:01.784107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.784068    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:01.784217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.784144    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:01.784217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:01.784179    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:02.295209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.290517    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"]
Oct 30 16:52:02.306343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.306311    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"]
Oct 30 16:52:02.310306 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.307526    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:02.323516 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.323485    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-07cb3c\""
Oct 30 16:52:02.332616 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.332569    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-07cb3c-dockercfg-hv5x7\""
Oct 30 16:52:02.404131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.404082    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdf5s\" (UniqueName: \"kubernetes.io/projected/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-kube-api-access-bdf5s\") pod \"payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp\" (UID: \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:02.410210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.404477    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp\" (UID: \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:02.505695 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.505659    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdf5s\" (UniqueName: \"kubernetes.io/projected/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-kube-api-access-bdf5s\") pod \"payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp\" (UID: \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:02.505890 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.505745    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp\" (UID: \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:02.506473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.506452    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp\" (UID: \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:02.526607 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.526572    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdf5s\" (UniqueName: \"kubernetes.io/projected/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-kube-api-access-bdf5s\") pod \"payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp\" (UID: \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:02.631729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.631655    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:02.734511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.734468    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" event={"ID":"22350f1f-626e-41f4-be41-aa75b103daea","Type":"ContainerStarted","Data":"d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235"}
Oct 30 16:52:02.734762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.734740    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" event={"ID":"22350f1f-626e-41f4-be41-aa75b103daea","Type":"ContainerStarted","Data":"52e9a202ef48d259923023396365b83f50df06cd631e55373430f6574c9b5cc1"}
Oct 30 16:52:02.962423 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.962312    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podStartSLOduration=1.962289653 podStartE2EDuration="1.962289653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:02.802552324 +0000 UTC m=+1752.656430590" watchObservedRunningTime="2026-10-30 16:52:02.962289653 +0000 UTC m=+1752.816167937"
Oct 30 16:52:02.963993 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:02.963965    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"]
Oct 30 16:52:02.969089 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:02.969059    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a13e5cb_1b3b_47fe_8e1f_22cb1cf8e592.slice/crio-e65f3802001da9d500fd75c84d7a19cad9a41cd62a9af74069613fb4732d4624 WatchSource:0}: Error finding container e65f3802001da9d500fd75c84d7a19cad9a41cd62a9af74069613fb4732d4624: Status 404 returned error can't find the container with id e65f3802001da9d500fd75c84d7a19cad9a41cd62a9af74069613fb4732d4624
Oct 30 16:52:03.441848 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:03.441811    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:03.443460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:03.443435    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 16:52:03.443595 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:03.443484    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 16:52:03.741938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:03.741850    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp" event={"ID":"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592","Type":"ContainerStarted","Data":"66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f"}
Oct 30 16:52:03.741938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:03.741904    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp" event={"ID":"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592","Type":"ContainerStarted","Data":"e65f3802001da9d500fd75c84d7a19cad9a41cd62a9af74069613fb4732d4624"}
Oct 30 16:52:03.741938 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:03.741932    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:03.796655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:03.788558    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp" podStartSLOduration=1.788534291 podStartE2EDuration="1.788534291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:03.786585442 +0000 UTC m=+1753.640463713" watchObservedRunningTime="2026-10-30 16:52:03.788534291 +0000 UTC m=+1753.642412556"
Oct 30 16:52:04.442077 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:04.442038    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 16:52:04.442510 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:04.442094    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 16:52:05.442336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:05.442301    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 16:52:05.442925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:05.442359    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 16:52:06.442793 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:06.442760    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 16:52:06.443205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:06.442814    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 16:52:07.453655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:07.453612    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 16:52:07.454355 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:07.454318    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 16:52:08.198213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:08.182226    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:08.374572 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:08.373315    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 16:52:08.446637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:08.443597    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 16:52:08.446637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:08.443662    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 16:52:08.583578 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:08.582304    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:08.618545 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:08.613714    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:09.447666 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:09.447620    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 16:52:09.447888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:09.447699    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 16:52:11.329402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:11.329366    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:11.330509 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:11.330482    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:11.331492 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:11.331467    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:52:12.603856 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.603817    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"]
Oct 30 16:52:12.608048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.608029    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:12.622148 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.622097    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4a8ab1\""
Oct 30 16:52:12.631629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.631602    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4a8ab1-dockercfg-6852c\""
Oct 30 16:52:12.642645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.642613    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"]
Oct 30 16:52:12.664001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.663964    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589db689-6c30-44f3-9c9f-b0c36ef2f063-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq\" (UID: \"589db689-6c30-44f3-9c9f-b0c36ef2f063\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:12.664244 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.664084    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjqr\" (UniqueName: \"kubernetes.io/projected/589db689-6c30-44f3-9c9f-b0c36ef2f063-kube-api-access-fqjqr\") pod \"payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq\" (UID: \"589db689-6c30-44f3-9c9f-b0c36ef2f063\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:12.700208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.699130    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"]
Oct 30 16:52:12.703770 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.703737    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:12.707622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.707593    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"]
Oct 30 16:52:12.766358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.766316    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589db689-6c30-44f3-9c9f-b0c36ef2f063-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq\" (UID: \"589db689-6c30-44f3-9c9f-b0c36ef2f063\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:12.766632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.766612    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrl8\" (UniqueName: \"kubernetes.io/projected/c6cde465-ffd3-4c14-b970-f9d98e344ff9-kube-api-access-4xrl8\") pod \"payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk\" (UID: \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:12.766841 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.766826    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjqr\" (UniqueName: \"kubernetes.io/projected/589db689-6c30-44f3-9c9f-b0c36ef2f063-kube-api-access-fqjqr\") pod \"payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq\" (UID: \"589db689-6c30-44f3-9c9f-b0c36ef2f063\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:12.767068 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.767051    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6cde465-ffd3-4c14-b970-f9d98e344ff9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk\" (UID: \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:12.768201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.768157    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589db689-6c30-44f3-9c9f-b0c36ef2f063-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq\" (UID: \"589db689-6c30-44f3-9c9f-b0c36ef2f063\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:12.790373 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.790338    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjqr\" (UniqueName: \"kubernetes.io/projected/589db689-6c30-44f3-9c9f-b0c36ef2f063-kube-api-access-fqjqr\") pod \"payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq\" (UID: \"589db689-6c30-44f3-9c9f-b0c36ef2f063\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:12.868624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.868535    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrl8\" (UniqueName: \"kubernetes.io/projected/c6cde465-ffd3-4c14-b970-f9d98e344ff9-kube-api-access-4xrl8\") pod \"payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk\" (UID: \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:12.868809 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.868642    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6cde465-ffd3-4c14-b970-f9d98e344ff9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk\" (UID: \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:12.869315 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.869296    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6cde465-ffd3-4c14-b970-f9d98e344ff9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk\" (UID: \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:12.889288 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.889252    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrl8\" (UniqueName: \"kubernetes.io/projected/c6cde465-ffd3-4c14-b970-f9d98e344ff9-kube-api-access-4xrl8\") pod \"payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk\" (UID: \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:12.924457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:12.924423    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:13.023707 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.023629    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:13.225458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.225382    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"]
Oct 30 16:52:13.228831 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:13.228796    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589db689_6c30_44f3_9c9f_b0c36ef2f063.slice/crio-aba513934d49719630a19690cf436e0aa47cf062573f6f6ecf00a01dad153e9f WatchSource:0}: Error finding container aba513934d49719630a19690cf436e0aa47cf062573f6f6ecf00a01dad153e9f: Status 404 returned error can't find the container with id aba513934d49719630a19690cf436e0aa47cf062573f6f6ecf00a01dad153e9f
Oct 30 16:52:13.341515 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.341488    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"]
Oct 30 16:52:13.344314 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:13.344289    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6cde465_ffd3_4c14_b970_f9d98e344ff9.slice/crio-186e7289aa3687cd37e36f39e9b91a792ec134302b5d3d8894a151e60e788f8f WatchSource:0}: Error finding container 186e7289aa3687cd37e36f39e9b91a792ec134302b5d3d8894a151e60e788f8f: Status 404 returned error can't find the container with id 186e7289aa3687cd37e36f39e9b91a792ec134302b5d3d8894a151e60e788f8f
Oct 30 16:52:13.869601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.869566    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk" event={"ID":"c6cde465-ffd3-4c14-b970-f9d98e344ff9","Type":"ContainerStarted","Data":"fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf"}
Oct 30 16:52:13.870058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.869620    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk" event={"ID":"c6cde465-ffd3-4c14-b970-f9d98e344ff9","Type":"ContainerStarted","Data":"186e7289aa3687cd37e36f39e9b91a792ec134302b5d3d8894a151e60e788f8f"}
Oct 30 16:52:13.870058 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.869652    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:13.871151 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.871124    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq" event={"ID":"589db689-6c30-44f3-9c9f-b0c36ef2f063","Type":"ContainerStarted","Data":"4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58"}
Oct 30 16:52:13.871255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.871175    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq" event={"ID":"589db689-6c30-44f3-9c9f-b0c36ef2f063","Type":"ContainerStarted","Data":"aba513934d49719630a19690cf436e0aa47cf062573f6f6ecf00a01dad153e9f"}
Oct 30 16:52:13.871255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.871232    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:13.907137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.907076    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk" podStartSLOduration=1.907059031 podStartE2EDuration="1.907059031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:13.905245728 +0000 UTC m=+1763.759123989" watchObservedRunningTime="2026-10-30 16:52:13.907059031 +0000 UTC m=+1763.760937296"
Oct 30 16:52:13.941778 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:13.941714    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq" podStartSLOduration=1.941699733 podStartE2EDuration="1.941699733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:13.940156845 +0000 UTC m=+1763.794035109" watchObservedRunningTime="2026-10-30 16:52:13.941699733 +0000 UTC m=+1763.795577997"
Oct 30 16:52:14.751468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:14.751438    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:52:21.916181 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:52:21.916145    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:52:24.880000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:24.879968    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:52:24.880454 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:24.880046    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:52:26.515569 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.515533    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"]
Oct 30 16:52:26.521095 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.521070    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.536534 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.536509    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4a8ab1-serving-cert\""
Oct 30 16:52:26.549930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.549898    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"]
Oct 30 16:52:26.649251 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.649218    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjc89\" (UniqueName: \"kubernetes.io/projected/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-kube-api-access-gjc89\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.649467 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.649262    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-metrics-certs\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.649512 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.649482    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.649576 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.649541    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.750438 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.750399    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.750717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.750460    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.750717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.750511    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjc89\" (UniqueName: \"kubernetes.io/projected/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-kube-api-access-gjc89\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.750717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.750535    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-metrics-certs\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.751200 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.751157    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.753056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.753035    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-metrics-certs\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.753128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.753063    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.771524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.771459    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjc89\" (UniqueName: \"kubernetes.io/projected/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-kube-api-access-gjc89\") pod \"maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:26.838425 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:26.838390    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:27.183773 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:27.183747    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"]
Oct 30 16:52:27.186990 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:27.186957    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b2692b_f6c7_47e0_88ce_7c57c9213bfe.slice/crio-a4c2da1e8ba450634c5032d4466594ad1f9c4656644da3d21786bb5f107d747d WatchSource:0}: Error finding container a4c2da1e8ba450634c5032d4466594ad1f9c4656644da3d21786bb5f107d747d: Status 404 returned error can't find the container with id a4c2da1e8ba450634c5032d4466594ad1f9c4656644da3d21786bb5f107d747d
Oct 30 16:52:27.963240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:27.963119    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw" event={"ID":"71b2692b-f6c7-47e0-88ce-7c57c9213bfe","Type":"ContainerStarted","Data":"38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf"}
Oct 30 16:52:27.963240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:27.963209    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw" event={"ID":"71b2692b-f6c7-47e0-88ce-7c57c9213bfe","Type":"ContainerStarted","Data":"a4c2da1e8ba450634c5032d4466594ad1f9c4656644da3d21786bb5f107d747d"}
Oct 30 16:52:27.963747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:27.963317    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:28.049209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:28.043864    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw" podStartSLOduration=1.6770875840000001 podStartE2EDuration="2.043842644s" totalImagesPullingTime="366.75506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:28.038844125 +0000 UTC m=+1777.892722401" watchObservedRunningTime="2026-10-30 16:52:28.043842644 +0000 UTC m=+1777.897720910"
Oct 30 16:52:29.491219 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:29.490288    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:29.620213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:29.615385    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:29.647213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:29.641324    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:30.107218 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.104042    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:30.131213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.126285    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:30.231246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.215364    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:30.298212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.284290    4497 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:30.389266 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.388707    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:30.640268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.640038    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:30.661658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.645842    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:52:30.661658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.646047    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:30.668257 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.663585    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:52:30.785210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.780107    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"]
Oct 30 16:52:30.956755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.954726    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"]
Oct 30 16:52:30.956755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.956285    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957277 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957235    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957287    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957320    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957370    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957396    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6abaa3-2dfe-4e36-bb38-700052316a62-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957520    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957572    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957683    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.957810 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.957712    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsjj\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-kube-api-access-rnsjj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:30.969823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.969741    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 16:52:30.969823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.969790    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"]
Oct 30 16:52:30.969968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.969830    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"]
Oct 30 16:52:30.969968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.969884    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 16:52:30.970146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.969976    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:30.970771 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:30.970745    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2d365c-openshift-default-dockercfg-gctmp\""
Oct 30 16:52:31.058466 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058431    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.058680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058476    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.058680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058504    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.058680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058524    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.058680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058544    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.058680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058655    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.058990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058710    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6abaa3-2dfe-4e36-bb38-700052316a62-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.058990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058788    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9zbr\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-kube-api-access-w9zbr\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.058990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058815    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.058990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058851    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.058990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058878    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.058990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058931    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.058990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.058972    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.059375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059003    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.059375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059056    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.059375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059105    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.059375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059147    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.059375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059209    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b5a92-0589-4e86-afba-6daf4b04ebd4-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.059375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059277    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.059375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059326    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.059375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059358    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsjj\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-kube-api-access-rnsjj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.059726 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059385    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6abaa3-2dfe-4e36-bb38-700052316a62-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.059726 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.059444    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.061227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.061207    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.061348 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.061288    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.106870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.106828    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.107897 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.107863    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsjj\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-kube-api-access-rnsjj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.160933 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.160898    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.161121 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.160943    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.161121 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.161001    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b5a92-0589-4e86-afba-6daf4b04ebd4-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.161121 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.161043    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.161511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.161119    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.161511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.161155    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.161511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.161302    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9zbr\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-kube-api-access-w9zbr\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.161511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.161355    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.161511 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.161394    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.162588 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.162543    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.162863 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.162841    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b5a92-0589-4e86-afba-6daf4b04ebd4-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.162989 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.162970    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.163061 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.163018    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.163120 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.163078    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.163679 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.163660    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.164839 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.164820    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.202214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.192038    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9zbr\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-kube-api-access-w9zbr\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.202214 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.194726    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.329345 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.329311    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:31.332393 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.332371    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:31.704261 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.704218    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"]
Oct 30 16:52:31.710001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.709973    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"]
Oct 30 16:52:31.716429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.716393    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:31.716544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.716460    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:31.716544 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.716490    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:31.716648 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:31.716634    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325b5a92_0589_4e86_afba_6daf4b04ebd4.slice/crio-e94520e846129cf8928a7ded41ad4a206d2ef260a3e162ba12c5877a4e56c903 WatchSource:0}: Error finding container e94520e846129cf8928a7ded41ad4a206d2ef260a3e162ba12c5877a4e56c903: Status 404 returned error can't find the container with id e94520e846129cf8928a7ded41ad4a206d2ef260a3e162ba12c5877a4e56c903
Oct 30 16:52:31.718963 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.718937    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:31.719021 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.718991    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:31.719055 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:31.719019    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:32.128343 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.128307    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" event={"ID":"3c6abaa3-2dfe-4e36-bb38-700052316a62","Type":"ContainerStarted","Data":"7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4"}
Oct 30 16:52:32.130224 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.130196    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" event={"ID":"3c6abaa3-2dfe-4e36-bb38-700052316a62","Type":"ContainerStarted","Data":"da85e3004312d79ad53c031e1addb44305f27485c4525f358343caa7531e3d29"}
Oct 30 16:52:32.131126 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.131105    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" event={"ID":"325b5a92-0589-4e86-afba-6daf4b04ebd4","Type":"ContainerStarted","Data":"0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460"}
Oct 30 16:52:32.131416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.131390    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" event={"ID":"325b5a92-0589-4e86-afba-6daf4b04ebd4","Type":"ContainerStarted","Data":"e94520e846129cf8928a7ded41ad4a206d2ef260a3e162ba12c5877a4e56c903"}
Oct 30 16:52:32.232205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.230811    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podStartSLOduration=2.230795574 podStartE2EDuration="2.230795574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:32.230083996 +0000 UTC m=+1782.083962260" watchObservedRunningTime="2026-10-30 16:52:32.230795574 +0000 UTC m=+1782.084673840"
Oct 30 16:52:32.238213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.237911    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podStartSLOduration=2.237887965 podStartE2EDuration="2.237887965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:32.18667114 +0000 UTC m=+1782.040549409" watchObservedRunningTime="2026-10-30 16:52:32.237887965 +0000 UTC m=+1782.091766231"
Oct 30 16:52:32.329657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.329621    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:32.330900 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.330878    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 16:52:32.330958 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.330927    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 16:52:32.333080 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.333064    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:32.334099 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.334081    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 16:52:32.334142 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:32.334120    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 16:52:33.337628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:33.337590    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 16:52:33.338255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:33.338222    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 16:52:33.338796 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:33.338773    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 16:52:33.338953 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:33.338935    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 16:52:34.040454 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:34.040422    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:52:34.364483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:34.364120    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 16:52:34.364483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:34.364178    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 16:52:34.365001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:34.364914    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 16:52:34.365001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:34.364981    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 16:52:35.330111 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:35.330067    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 16:52:35.330444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:35.330131    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 16:52:35.333460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:35.333433    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 16:52:35.333589 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:35.333478    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 16:52:36.333272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:36.332995    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 16:52:36.333272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:36.333056    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 16:52:36.333888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:36.333492    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 16:52:36.333888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:36.333529    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 16:52:37.329811 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:37.329755    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 16:52:37.329982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:37.329835    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 16:52:37.333742 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:37.333714    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 16:52:37.334038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:37.333784    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 16:52:38.337210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:38.334673    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 16:52:38.337210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:38.334732    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 16:52:38.337210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:38.335123    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 16:52:38.337210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:38.335154    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 16:52:39.329849 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:39.329811    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 16:52:39.330056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:39.329877    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 16:52:39.333228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:39.333181    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 16:52:39.333362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:39.333253    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 16:52:40.249913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.249824    4497 generic.go:356] "Generic (PLEG): container finished" podID="42284a65-b5fb-4b79-b717-1961164e79ce" containerID="6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90" exitCode=0
Oct 30 16:52:40.249913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.249890    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" event={"ID":"42284a65-b5fb-4b79-b717-1961164e79ce","Type":"ContainerDied","Data":"6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90"}
Oct 30 16:52:40.250517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.250369    4497 scope.go:122] "RemoveContainer" containerID="6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90"
Oct 30 16:52:40.251711 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.251680    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:40.251783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.251757    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:40.251823 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.251789    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:40.346638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.343249    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:40.346638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.344569    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:40.349911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.348629    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:40.349911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.349302    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:52:40.349911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.349753    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:40.351304 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:40.351281    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:52:41.258439 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:41.258403    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" event={"ID":"42284a65-b5fb-4b79-b717-1961164e79ce","Type":"ContainerStarted","Data":"b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7"}
Oct 30 16:52:41.991394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:41.990545    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:52:41.997532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:41.997492    4497 patch_prober.go:29] interesting pod/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 16:52:41.997746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:41.997553    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 16:52:42.979176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:42.979142    4497 patch_prober.go:29] interesting pod/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 16:52:42.979895 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:42.979849    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 16:52:43.989614 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:43.989574    4497 patch_prober.go:29] interesting pod/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 16:52:43.990277 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:43.990242    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 16:52:44.712696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:44.712639    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:52:44.994686 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:44.994589    4497 patch_prober.go:29] interesting pod/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 16:52:44.995292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:44.995258    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 16:52:45.761717 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.761682    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"]
Oct 30 16:52:45.766148 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.766128    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.804577 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.804547    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"]
Oct 30 16:52:45.855581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.855548    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.855753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.855704    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsnn2\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-kube-api-access-wsnn2\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.855825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.855756    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.855825 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.855777    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.855907 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.855888    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.855955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.855918    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.855955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.855953    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a9d517-27d1-4c1b-b29a-1d413495d659-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.856061 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.856016    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.856096 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.856083    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957092    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957134    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957158    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957225    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a9d517-27d1-4c1b-b29a-1d413495d659-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957244    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957393    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957464    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957554    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957627    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsnn2\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-kube-api-access-wsnn2\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.957733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.957717    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.958038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.958013    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a9d517-27d1-4c1b-b29a-1d413495d659-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.958262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.958178    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.958386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.958257    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.958456 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.958330    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.959936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.959916    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.960059 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.960039    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:45.998220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.994896    4497 patch_prober.go:29] interesting pod/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 16:52:45.998220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.994952    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 16:52:45.999753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:45.999716    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:46.006549 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:46.006520    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsnn2\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-kube-api-access-wsnn2\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:46.113247 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:46.113140    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:46.435827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:46.435739    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"]
Oct 30 16:52:46.442420 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:46.442353    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a9d517_27d1_4c1b_b29a_1d413495d659.slice/crio-9795407efb6947e62eab65228b6564d7dd687606b2e648fd789cc9f92d0311b1 WatchSource:0}: Error finding container 9795407efb6947e62eab65228b6564d7dd687606b2e648fd789cc9f92d0311b1: Status 404 returned error can't find the container with id 9795407efb6947e62eab65228b6564d7dd687606b2e648fd789cc9f92d0311b1
Oct 30 16:52:46.445147 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:46.445110    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:46.445262 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:46.445225    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:46.445303 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:46.445259    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:46.982694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:46.982667    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:52:46.983751 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:46.983734    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:52:47.346208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.346157    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"]
Oct 30 16:52:47.368229 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.364126    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.391207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.387543    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-cb6251-openshift-default-dockercfg-5cbjg\""
Oct 30 16:52:47.391207 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.387758    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"]
Oct 30 16:52:47.420210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.415987    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" event={"ID":"b7a9d517-27d1-4c1b-b29a-1d413495d659","Type":"ContainerStarted","Data":"de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079"}
Oct 30 16:52:47.420210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.416047    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" event={"ID":"b7a9d517-27d1-4c1b-b29a-1d413495d659","Type":"ContainerStarted","Data":"9795407efb6947e62eab65228b6564d7dd687606b2e648fd789cc9f92d0311b1"}
Oct 30 16:52:47.469417 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.469351    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podStartSLOduration=2.469331914 podStartE2EDuration="2.469331914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:47.468729686 +0000 UTC m=+1797.322607953" watchObservedRunningTime="2026-10-30 16:52:47.469331914 +0000 UTC m=+1797.323210180"
Oct 30 16:52:47.490113 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.490065    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999975c1-3ece-490c-ad9f-a039410f320a-istio-podinfo\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.519209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.516156    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-data\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.519209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.516348    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-istio-token\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.519209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.517081    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-certs\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.519209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.517220    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-credential-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.519209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.517301    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.519209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.517342    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx57g\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-kube-api-access-rx57g\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.519209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.517881    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-envoy\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.519209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.517925    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999975c1-3ece-490c-ad9f-a039410f320a-istiod-ca-cert\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623035    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-certs\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623122    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-credential-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623164    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623206    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx57g\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-kube-api-access-rx57g\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623300    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-envoy\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623331    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999975c1-3ece-490c-ad9f-a039410f320a-istiod-ca-cert\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623383    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999975c1-3ece-490c-ad9f-a039410f320a-istio-podinfo\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623436    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-data\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.623783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.623501    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-istio-token\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.624279 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.624253    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-certs\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.624522 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.624504    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-credential-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.624729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.624710    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.625983 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.625953    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-data\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.626255 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.626232    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999975c1-3ece-490c-ad9f-a039410f320a-istiod-ca-cert\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.628620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.628584    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-envoy\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.637711 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.637680    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999975c1-3ece-490c-ad9f-a039410f320a-istio-podinfo\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.646201 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.644852    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-istio-token\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.648857 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.648839    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx57g\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-kube-api-access-rx57g\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:47.715846 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:47.715809    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:48.081264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.060113    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"]
Oct 30 16:52:48.081902 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:48.081856    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999975c1_3ece_490c_ad9f_a039410f320a.slice/crio-5ee624eaa354ec21d29a692e5ad00d407fcd54e1bef8cecc12d4de65d1c19d29 WatchSource:0}: Error finding container 5ee624eaa354ec21d29a692e5ad00d407fcd54e1bef8cecc12d4de65d1c19d29: Status 404 returned error can't find the container with id 5ee624eaa354ec21d29a692e5ad00d407fcd54e1bef8cecc12d4de65d1c19d29
Oct 30 16:52:48.112833 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.112789    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:48.112981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.112865    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:48.112981 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.112895    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:48.114699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.113342    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:48.115089 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.114970    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 16:52:48.115089 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.115020    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 16:52:48.424651 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.424558    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" event={"ID":"999975c1-3ece-490c-ad9f-a039410f320a","Type":"ContainerStarted","Data":"2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7"}
Oct 30 16:52:48.424651 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.424629    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" event={"ID":"999975c1-3ece-490c-ad9f-a039410f320a","Type":"ContainerStarted","Data":"5ee624eaa354ec21d29a692e5ad00d407fcd54e1bef8cecc12d4de65d1c19d29"}
Oct 30 16:52:48.510686 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.510631    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" podStartSLOduration=1.510616709 podStartE2EDuration="1.510616709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:48.50925431 +0000 UTC m=+1798.363132580" watchObservedRunningTime="2026-10-30 16:52:48.510616709 +0000 UTC m=+1798.364494973"
Oct 30 16:52:48.716391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:48.716306    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:49.114120 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:49.114084    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 16:52:49.114412 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:49.114146    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 16:52:49.722488 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:49.722457    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:50.113781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:50.113743    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 16:52:50.113967 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:50.113803    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 16:52:50.436767 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:50.436695    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:50.437730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:50.437709    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:52:51.114238 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:51.114196    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 16:52:51.114759 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:51.114267    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 16:52:51.921587 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:52:51.921550    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:52:52.113697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:52.113662    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 16:52:52.113898 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:52.113714    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 16:52:53.121211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:53.121066    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 16:52:53.121211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:53.121125    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 16:52:53.603735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:53.603306    4497 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:52:53.603735 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:53.603417    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:53.996866 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:53.996761    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"]
Oct 30 16:52:54.006222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.003602    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:54.012340 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.012067    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"]
Oct 30 16:52:54.026352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.018858    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-cb6251\""
Oct 30 16:52:54.031723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.031691    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-cb6251-dockercfg-r74wg\""
Oct 30 16:52:54.076223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.073403    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7vx\" (UniqueName: \"kubernetes.io/projected/45758dca-74d1-4578-8750-171b237c972c-kube-api-access-gr7vx\") pod \"payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds\" (UID: \"45758dca-74d1-4578-8750-171b237c972c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:54.076223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.073564    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45758dca-74d1-4578-8750-171b237c972c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds\" (UID: \"45758dca-74d1-4578-8750-171b237c972c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:54.166212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.141357    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 16:52:54.167375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.167327    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 16:52:54.176085 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.176033    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45758dca-74d1-4578-8750-171b237c972c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds\" (UID: \"45758dca-74d1-4578-8750-171b237c972c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:54.176497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.176476    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7vx\" (UniqueName: \"kubernetes.io/projected/45758dca-74d1-4578-8750-171b237c972c-kube-api-access-gr7vx\") pod \"payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds\" (UID: \"45758dca-74d1-4578-8750-171b237c972c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:54.215743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.178306    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45758dca-74d1-4578-8750-171b237c972c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds\" (UID: \"45758dca-74d1-4578-8750-171b237c972c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:54.291211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.287363    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7vx\" (UniqueName: \"kubernetes.io/projected/45758dca-74d1-4578-8750-171b237c972c-kube-api-access-gr7vx\") pod \"payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds\" (UID: \"45758dca-74d1-4578-8750-171b237c972c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:54.402605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:54.402566    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:55.106356 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.106303    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:55.338199 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.338130    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:55.495267 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.494549    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:55.603222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.564390    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"]
Oct 30 16:52:55.635212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.609357    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"]
Oct 30 16:52:55.660211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.610337    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.685210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.680506    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-dd715e-openshift-default-dockercfg-fhwwv\""
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.699982    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0fca89-11f6-4450-9b7b-27f9fe014668-istiod-ca-cert\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.700068    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-certs\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.700156    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-envoy\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.700207    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r467s\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-kube-api-access-r467s\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.700242    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-data\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.700272    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-token\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.700313    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-credential-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.700355    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-podinfo\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.750629 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.700403    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.967216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.963297    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-credential-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.974216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.967511    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-podinfo\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:55.974534 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.974502    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.019275 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.014544    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 16:52:56.020444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.020413    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-credential-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.025539    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:55.983760    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"]
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.058919    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0fca89-11f6-4450-9b7b-27f9fe014668-istiod-ca-cert\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.069037    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-certs\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.069154    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-envoy\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.069212    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r467s\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-kube-api-access-r467s\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.069256    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-data\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.069290    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-token\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.059927    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0fca89-11f6-4450-9b7b-27f9fe014668-istiod-ca-cert\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.109692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.070069    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-certs\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.134641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.131475    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 16:52:56.153298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.141855    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:52:56.170781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.167688    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-podinfo\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.170781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.169735    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-data\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.220264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.172320    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 16:52:56.220264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.172383    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 16:52:56.240563 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.238951    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 16:52:56.256655 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.256597    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-token\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.257064 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.257042    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-envoy\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.291108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.288840    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r467s\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-kube-api-access-r467s\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.370770 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.352316    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 16:52:56.487239 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.487019    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:56.488223 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.487880    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:56.488762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.488741    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:52:56.565250 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.565139    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:56.896479 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.896371    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"]
Oct 30 16:52:56.899393 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:52:56.899366    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0fca89_11f6_4450_9b7b_27f9fe014668.slice/crio-122538d0e3084161941dd904b7222b6da8dbcca76f9a4a4cef86a2a87ed63420 WatchSource:0}: Error finding container 122538d0e3084161941dd904b7222b6da8dbcca76f9a4a4cef86a2a87ed63420: Status 404 returned error can't find the container with id 122538d0e3084161941dd904b7222b6da8dbcca76f9a4a4cef86a2a87ed63420
Oct 30 16:52:56.902341 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.902306    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:56.902429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.902375    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:56.902429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:56.902408    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:52:57.338212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:57.336163    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds" event={"ID":"45758dca-74d1-4578-8750-171b237c972c","Type":"ContainerStarted","Data":"b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2"}
Oct 30 16:52:57.338212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:57.336279    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds" event={"ID":"45758dca-74d1-4578-8750-171b237c972c","Type":"ContainerStarted","Data":"73b05b78b19d62cef92a17e719923c7c185d127d1f3d9d277ee761b2ebab9d12"}
Oct 30 16:52:57.338212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:57.338029    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:52:57.341018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:57.340175    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" event={"ID":"0a0fca89-11f6-4450-9b7b-27f9fe014668","Type":"ContainerStarted","Data":"f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b"}
Oct 30 16:52:57.341018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:57.340248    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" event={"ID":"0a0fca89-11f6-4450-9b7b-27f9fe014668","Type":"ContainerStarted","Data":"122538d0e3084161941dd904b7222b6da8dbcca76f9a4a4cef86a2a87ed63420"}
Oct 30 16:52:57.379216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:57.377050    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds" podStartSLOduration=4.377035156 podStartE2EDuration="4.377035156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:57.374001187 +0000 UTC m=+1807.227879479" watchObservedRunningTime="2026-10-30 16:52:57.377035156 +0000 UTC m=+1807.230913421"
Oct 30 16:52:57.420286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:57.419899    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" podStartSLOduration=2.419877953 podStartE2EDuration="2.419877953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:57.41551237 +0000 UTC m=+1807.269390647" watchObservedRunningTime="2026-10-30 16:52:57.419877953 +0000 UTC m=+1807.273756218"
Oct 30 16:52:57.566122 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:57.566087    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:58.571325 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:58.571295    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:59.352527 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:59.352485    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:52:59.353416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:52:59.353391    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:53:02.326388 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:02.326353    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"]
Oct 30 16:53:02.373637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:02.373604    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"]
Oct 30 16:53:07.326998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.326951    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h" podUID="ce38fe36-c59b-4a43-97f3-3a55140a28e1" containerName="payload-processing" containerID="cri-o://34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18" gracePeriod=25
Oct 30 16:53:07.373969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.373929    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh" podUID="9361fbab-d01a-426c-9669-589c357bb916" containerName="payload-pre-processing" containerID="cri-o://5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425" gracePeriod=25
Oct 30 16:53:07.659648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.659619    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:53:07.688637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.688605    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:53:07.744359 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.744322    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9361fbab-d01a-426c-9669-589c357bb916-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9361fbab-d01a-426c-9669-589c357bb916-plugins-config-volume\") pod \"9361fbab-d01a-426c-9669-589c357bb916\" (UID: \"9361fbab-d01a-426c-9669-589c357bb916\") "
Oct 30 16:53:07.744552 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.744420    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce38fe36-c59b-4a43-97f3-3a55140a28e1-kube-api-access-drp9h\" (UniqueName: \"kubernetes.io/projected/ce38fe36-c59b-4a43-97f3-3a55140a28e1-kube-api-access-drp9h\") pod \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\" (UID: \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\") "
Oct 30 16:53:07.744646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.744628    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9361fbab-d01a-426c-9669-589c357bb916-kube-api-access-8kx49\" (UniqueName: \"kubernetes.io/projected/9361fbab-d01a-426c-9669-589c357bb916-kube-api-access-8kx49\") pod \"9361fbab-d01a-426c-9669-589c357bb916\" (UID: \"9361fbab-d01a-426c-9669-589c357bb916\") "
Oct 30 16:53:07.744827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.744719    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce38fe36-c59b-4a43-97f3-3a55140a28e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38fe36-c59b-4a43-97f3-3a55140a28e1-plugins-config-volume\") pod \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\" (UID: \"ce38fe36-c59b-4a43-97f3-3a55140a28e1\") "
Oct 30 16:53:07.744827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.744773    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9361fbab-d01a-426c-9669-589c357bb916-plugins-config-volume" pod "9361fbab-d01a-426c-9669-589c357bb916" (UID: "9361fbab-d01a-426c-9669-589c357bb916"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:53:07.745297 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.745274    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9361fbab-d01a-426c-9669-589c357bb916-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:07.746053 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.746025    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce38fe36-c59b-4a43-97f3-3a55140a28e1-plugins-config-volume" pod "ce38fe36-c59b-4a43-97f3-3a55140a28e1" (UID: "ce38fe36-c59b-4a43-97f3-3a55140a28e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:53:07.747480 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.747456    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9361fbab-d01a-426c-9669-589c357bb916-kube-api-access-8kx49" pod "9361fbab-d01a-426c-9669-589c357bb916" (UID: "9361fbab-d01a-426c-9669-589c357bb916"). InnerVolumeSpecName "kube-api-access-8kx49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:53:07.747780 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.747753    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce38fe36-c59b-4a43-97f3-3a55140a28e1-kube-api-access-drp9h" pod "ce38fe36-c59b-4a43-97f3-3a55140a28e1" (UID: "ce38fe36-c59b-4a43-97f3-3a55140a28e1"). InnerVolumeSpecName "kube-api-access-drp9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:53:07.846851 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.846810    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38fe36-c59b-4a43-97f3-3a55140a28e1-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:07.846851 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.846844    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drp9h\" (UniqueName: \"kubernetes.io/projected/ce38fe36-c59b-4a43-97f3-3a55140a28e1-kube-api-access-drp9h\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:07.846851 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:07.846854    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kx49\" (UniqueName: \"kubernetes.io/projected/9361fbab-d01a-426c-9669-589c357bb916-kube-api-access-8kx49\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:08.414983 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.414944    4497 generic.go:356] "Generic (PLEG): container finished" podID="ce38fe36-c59b-4a43-97f3-3a55140a28e1" containerID="34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18" exitCode=0
Oct 30 16:53:08.415473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.414998    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"
Oct 30 16:53:08.415473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.415038    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h" event={"ID":"ce38fe36-c59b-4a43-97f3-3a55140a28e1","Type":"ContainerDied","Data":"34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18"}
Oct 30 16:53:08.415473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.415094    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h" event={"ID":"ce38fe36-c59b-4a43-97f3-3a55140a28e1","Type":"ContainerDied","Data":"0cdc277a9f3e4adc2cbba330344fcda3b4f7928847545b7d5194757346a8030c"}
Oct 30 16:53:08.415473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.415123    4497 scope.go:122] "RemoveContainer" containerID="34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18"
Oct 30 16:53:08.416517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.416495    4497 generic.go:356] "Generic (PLEG): container finished" podID="9361fbab-d01a-426c-9669-589c357bb916" containerID="5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425" exitCode=0
Oct 30 16:53:08.416596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.416531    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"
Oct 30 16:53:08.416596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.416578    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh" event={"ID":"9361fbab-d01a-426c-9669-589c357bb916","Type":"ContainerDied","Data":"5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425"}
Oct 30 16:53:08.416699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.416632    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh" event={"ID":"9361fbab-d01a-426c-9669-589c357bb916","Type":"ContainerDied","Data":"6e5748ce237b91a48e20d6f884bb889d0dcd5d0cc914346c2c8b9e9d6bd4c7ff"}
Oct 30 16:53:08.424911 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.424895    4497 scope.go:122] "RemoveContainer" containerID="34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18"
Oct 30 16:53:08.425224 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:08.425167    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18\": container with ID starting with 34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18 not found: ID does not exist" containerID="34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18"
Oct 30 16:53:08.425324 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.425230    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18"} err="failed to get container status \"34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18\": rpc error: code = NotFound desc = could not find container \"34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18\": container with ID starting with 34b483d79ee6966a90e698e2f830a6c941f99a3b7f65c54644682f0e9c0a1c18 not found: ID does not exist"
Oct 30 16:53:08.425324 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.425247    4497 scope.go:122] "RemoveContainer" containerID="5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425"
Oct 30 16:53:08.434357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.434340    4497 scope.go:122] "RemoveContainer" containerID="5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425"
Oct 30 16:53:08.434623 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:08.434604    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425\": container with ID starting with 5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425 not found: ID does not exist" containerID="5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425"
Oct 30 16:53:08.434671 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.434631    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425"} err="failed to get container status \"5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425\": rpc error: code = NotFound desc = could not find container \"5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425\": container with ID starting with 5af5e9bf5ee43f1344fa98ca1146f988281cca2f06387567ff3e873bbfd22425 not found: ID does not exist"
Oct 30 16:53:08.459859 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.459832    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"]
Oct 30 16:53:08.469107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.469080    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-52f3fd-69b74bb756-7xc2h"]
Oct 30 16:53:08.492674 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.492643    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"]
Oct 30 16:53:08.498620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:08.498595    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-52f3fd-7f65548b9f-c4lsh"]
Oct 30 16:53:09.355912 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.355885    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 16:53:09.939809 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.939769    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"]
Oct 30 16:53:09.940444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.940427    4497 cpu_manager.go:414] "containerMap: removing container" podUID="9361fbab-d01a-426c-9669-589c357bb916" containerName="payload-pre-processing"
Oct 30 16:53:09.940497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.940446    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9361fbab-d01a-426c-9669-589c357bb916" containerName="payload-pre-processing"
Oct 30 16:53:09.940497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.940479    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ce38fe36-c59b-4a43-97f3-3a55140a28e1" containerName="payload-processing"
Oct 30 16:53:09.940497 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.940485    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce38fe36-c59b-4a43-97f3-3a55140a28e1" containerName="payload-processing"
Oct 30 16:53:09.940603 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.940593    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce38fe36-c59b-4a43-97f3-3a55140a28e1" containerName="payload-processing"
Oct 30 16:53:09.940638 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.940610    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="9361fbab-d01a-426c-9669-589c357bb916" containerName="payload-pre-processing"
Oct 30 16:53:09.945581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.945560    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:09.960961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.960930    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-77f4f4-serving-cert\""
Oct 30 16:53:09.973061 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.973025    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:09.973240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.973067    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rjq\" (UniqueName: \"kubernetes.io/projected/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-kube-api-access-77rjq\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:09.973240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.973159    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-metrics-certs\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:09.973415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.973398    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:09.985322 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:09.985289    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"]
Oct 30 16:53:10.074654 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.074612    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.074821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.074686    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.074821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.074710    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rjq\" (UniqueName: \"kubernetes.io/projected/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-kube-api-access-77rjq\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.074821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.074739    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-metrics-certs\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.074821 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:10.074753    4497 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-serving-cert: secret "maas-api-e2e-praxis-b-77f4f4-serving-cert" not found
Oct 30 16:53:10.075027 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:10.074822    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls podName:651fa5cf-eaa0-4fb8-965e-48bd9564d79b nodeName:}" failed. No retries permitted until 2026-10-30 16:53:10.57480109 +0000 UTC m=+1820.428679347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls") pod "maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh" (UID: "651fa5cf-eaa0-4fb8-965e-48bd9564d79b") : secret "maas-api-e2e-praxis-b-77f4f4-serving-cert" not found
Oct 30 16:53:10.075396 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.075373    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.077310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.077288    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-metrics-certs\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.101976 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.101948    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rjq\" (UniqueName: \"kubernetes.io/projected/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-kube-api-access-77rjq\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.308521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.308490    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9361fbab-d01a-426c-9669-589c357bb916" path="/var/lib/kubelet/pods/9361fbab-d01a-426c-9669-589c357bb916/volumes"
Oct 30 16:53:10.308945 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.308930    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce38fe36-c59b-4a43-97f3-3a55140a28e1" path="/var/lib/kubelet/pods/ce38fe36-c59b-4a43-97f3-3a55140a28e1/volumes"
Oct 30 16:53:10.582718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.582623    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.585230 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.585161    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:10.864678 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:10.864589    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:11.174215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:11.174154    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"]
Oct 30 16:53:11.177051 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:53:11.177023    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651fa5cf_eaa0_4fb8_965e_48bd9564d79b.slice/crio-c45abe2153d94fabf084e74b1d3fb8d551cf2c3aa2ad8abf5656e1353005ca67 WatchSource:0}: Error finding container c45abe2153d94fabf084e74b1d3fb8d551cf2c3aa2ad8abf5656e1353005ca67: Status 404 returned error can't find the container with id c45abe2153d94fabf084e74b1d3fb8d551cf2c3aa2ad8abf5656e1353005ca67
Oct 30 16:53:11.440109 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:11.440019    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh" event={"ID":"651fa5cf-eaa0-4fb8-965e-48bd9564d79b","Type":"ContainerStarted","Data":"c45abe2153d94fabf084e74b1d3fb8d551cf2c3aa2ad8abf5656e1353005ca67"}
Oct 30 16:53:12.448440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:12.448400    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh" event={"ID":"651fa5cf-eaa0-4fb8-965e-48bd9564d79b","Type":"ContainerStarted","Data":"799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb"}
Oct 30 16:53:12.448869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:12.448485    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:12.510433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:12.510380    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh" podStartSLOduration=3.173162019 podStartE2EDuration="3.510365161s" totalImagesPullingTime="337.203142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:12.506599726 +0000 UTC m=+1822.360477991" watchObservedRunningTime="2026-10-30 16:53:12.510365161 +0000 UTC m=+1822.364243425"
Oct 30 16:53:15.879826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:15.879788    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"]
Oct 30 16:53:15.884040 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:15.884019    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:15.910826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:15.910785    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"]
Oct 30 16:53:16.022273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022233    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.022468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022327    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.022468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022394    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.022468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022433    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbvsn\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-kube-api-access-nbvsn\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.022593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022512    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.022593 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022545    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.022661 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022634    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.022710 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022698    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.022764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.022749    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.123936 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.123896    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.123993    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124050    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124145    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124178    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124363 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124253    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124363 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124329    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbvsn\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-kube-api-access-nbvsn\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124474 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124364    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124474 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124402    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124474 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124403    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124521    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124776 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124757    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.124872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.124805    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.125032 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.125006    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.126312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.126287    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.126898 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.126881    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.144758 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.144691    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.145503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.145482    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbvsn\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-kube-api-access-nbvsn\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.204972 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.204933    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:16.535382 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.535353    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"]
Oct 30 16:53:16.536793 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:53:16.536767    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ef69a4_f3c8_46ba_9de1_b151e7837edb.slice/crio-3df195b4ed7b9d999072b5d328c2d8f285b2418b981cee7c9c7779bed7da384a WatchSource:0}: Error finding container 3df195b4ed7b9d999072b5d328c2d8f285b2418b981cee7c9c7779bed7da384a: Status 404 returned error can't find the container with id 3df195b4ed7b9d999072b5d328c2d8f285b2418b981cee7c9c7779bed7da384a
Oct 30 16:53:16.539362 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.539331    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:16.539434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.539397    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:16.539434 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:16.539427    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:17.549339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:17.549289    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" event={"ID":"19ef69a4-f3c8-46ba-9de1-b151e7837edb","Type":"ContainerStarted","Data":"ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a"}
Oct 30 16:53:17.553628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:17.553602    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" event={"ID":"19ef69a4-f3c8-46ba-9de1-b151e7837edb","Type":"ContainerStarted","Data":"3df195b4ed7b9d999072b5d328c2d8f285b2418b981cee7c9c7779bed7da384a"}
Oct 30 16:53:17.614061 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:17.613993    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" podStartSLOduration=2.613971841 podStartE2EDuration="2.613971841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:17.6016807 +0000 UTC m=+1827.455558966" watchObservedRunningTime="2026-10-30 16:53:17.613971841 +0000 UTC m=+1827.467850107"
Oct 30 16:53:18.205358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:18.205309    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:18.206991 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:18.206965    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 16:53:18.207146 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:18.207014    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 16:53:18.509274 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:18.504401    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:53:19.205652 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:19.205618    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 16:53:19.206098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:19.205673    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 16:53:20.205278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:20.205245    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 16:53:20.205483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:20.205306    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 16:53:21.002786 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.002747    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"]
Oct 30 16:53:21.003329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.002981    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8" podUID="a0d2a86d-4ccb-453f-a07a-48af87c66e38" containerName="maas-api" containerID="cri-o://f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c" gracePeriod=30
Oct 30 16:53:21.205385 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.205301    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 16:53:21.205385 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.205364    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 16:53:21.414748 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.414703    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:53:21.554811 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.554773    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-metrics-certs\") pod \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") "
Oct 30 16:53:21.555677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.555652    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d2a86d-4ccb-453f-a07a-48af87c66e38-kube-api-access-fnxzs\" (UniqueName: \"kubernetes.io/projected/a0d2a86d-4ccb-453f-a07a-48af87c66e38-kube-api-access-fnxzs\") pod \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") "
Oct 30 16:53:21.556258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.556237    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d2a86d-4ccb-453f-a07a-48af87c66e38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d2a86d-4ccb-453f-a07a-48af87c66e38-openshift-service-ca\") pod \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") "
Oct 30 16:53:21.560519 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.560389    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d2a86d-4ccb-453f-a07a-48af87c66e38-openshift-service-ca" pod "a0d2a86d-4ccb-453f-a07a-48af87c66e38" (UID: "a0d2a86d-4ccb-453f-a07a-48af87c66e38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:53:21.560719 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.560700    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-maas-api-tls\") pod \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\" (UID: \"a0d2a86d-4ccb-453f-a07a-48af87c66e38\") "
Oct 30 16:53:21.561845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.561822    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d2a86d-4ccb-453f-a07a-48af87c66e38-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:21.618571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.618361    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-maas-api-tls" pod "a0d2a86d-4ccb-453f-a07a-48af87c66e38" (UID: "a0d2a86d-4ccb-453f-a07a-48af87c66e38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:53:21.632529 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.632459    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-metrics-certs" pod "a0d2a86d-4ccb-453f-a07a-48af87c66e38" (UID: "a0d2a86d-4ccb-453f-a07a-48af87c66e38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:53:21.649870 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.649755    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d2a86d-4ccb-453f-a07a-48af87c66e38-kube-api-access-fnxzs" pod "a0d2a86d-4ccb-453f-a07a-48af87c66e38" (UID: "a0d2a86d-4ccb-453f-a07a-48af87c66e38"). InnerVolumeSpecName "kube-api-access-fnxzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:53:21.653592 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.653557    4497 generic.go:356] "Generic (PLEG): container finished" podID="a0d2a86d-4ccb-453f-a07a-48af87c66e38" containerID="f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c" exitCode=0
Oct 30 16:53:21.653728 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.653617    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8" event={"ID":"a0d2a86d-4ccb-453f-a07a-48af87c66e38","Type":"ContainerDied","Data":"f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c"}
Oct 30 16:53:21.653728 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.653669    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8" event={"ID":"a0d2a86d-4ccb-453f-a07a-48af87c66e38","Type":"ContainerDied","Data":"26d1c79420b00808ae4fe1848d1df4f3d7224442a644d89f0f0a0513a7070e69"}
Oct 30 16:53:21.653728 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.653703    4497 scope.go:122] "RemoveContainer" containerID="f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c"
Oct 30 16:53:21.653874 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.653854    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"
Oct 30 16:53:21.663622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.663596    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnxzs\" (UniqueName: \"kubernetes.io/projected/a0d2a86d-4ccb-453f-a07a-48af87c66e38-kube-api-access-fnxzs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:21.663721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.663630    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:21.663721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.663646    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0d2a86d-4ccb-453f-a07a-48af87c66e38-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:21.667831 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.667707    4497 scope.go:122] "RemoveContainer" containerID="f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c"
Oct 30 16:53:21.668246 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:21.668081    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c\": container with ID starting with f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c not found: ID does not exist" containerID="f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c"
Oct 30 16:53:21.668246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.668109    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c"} err="failed to get container status \"f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c\": rpc error: code = NotFound desc = could not find container \"f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c\": container with ID starting with f3cc00342d5b45e4c714a0672cf90f43c6ece8250c5ab3c69535df3ffebd057c not found: ID does not exist"
Oct 30 16:53:21.715903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.715871    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"]
Oct 30 16:53:21.727386 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.720634    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-52f3fd-79d6b75df8-khzq8"]
Oct 30 16:53:21.817249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.817210    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"]
Oct 30 16:53:21.817886 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.817871    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a0d2a86d-4ccb-453f-a07a-48af87c66e38" containerName="maas-api"
Oct 30 16:53:21.817932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.817888    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d2a86d-4ccb-453f-a07a-48af87c66e38" containerName="maas-api"
Oct 30 16:53:21.818002 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.817993    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d2a86d-4ccb-453f-a07a-48af87c66e38" containerName="maas-api"
Oct 30 16:53:21.822954 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.822938    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.835391 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.835362    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-dd715e-serving-cert\""
Oct 30 16:53:21.848031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.847953    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"]
Oct 30 16:53:21.866745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.866703    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.866898 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.866787    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.866944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.866893    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nd25\" (UniqueName: \"kubernetes.io/projected/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-kube-api-access-8nd25\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.866944 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.866928    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.933222 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:21.933166    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:53:21.968168 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.968132    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nd25\" (UniqueName: \"kubernetes.io/projected/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-kube-api-access-8nd25\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.968394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.968214    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.968460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.968441    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.968524 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.968508    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.969338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.969308    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.971041 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.971015    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.985215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.980255    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:21.999051 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:21.998981    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nd25\" (UniqueName: \"kubernetes.io/projected/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-kube-api-access-8nd25\") pod \"maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:22.053828 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.053796    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"]
Oct 30 16:53:22.058542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.058517    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:22.106134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.083510    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-cf4621\""
Oct 30 16:53:22.106134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.092295    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt\" (UID: \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:22.106134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.092473    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvkh\" (UniqueName: \"kubernetes.io/projected/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-kube-api-access-wfvkh\") pod \"payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt\" (UID: \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:22.109913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.109878    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"]
Oct 30 16:53:22.136811 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.136774    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:22.227218 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.223097    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt\" (UID: \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:22.227218 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.223243    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvkh\" (UniqueName: \"kubernetes.io/projected/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-kube-api-access-wfvkh\") pod \"payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt\" (UID: \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:22.306236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.241108    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt\" (UID: \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:22.306236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.241703    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 16:53:22.306236 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.241750    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 16:53:22.344211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.331275    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvkh\" (UniqueName: \"kubernetes.io/projected/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-kube-api-access-wfvkh\") pod \"payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt\" (UID: \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:22.366268 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.360382    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d2a86d-4ccb-453f-a07a-48af87c66e38" path="/var/lib/kubelet/pods/a0d2a86d-4ccb-453f-a07a-48af87c66e38/volumes"
Oct 30 16:53:22.428649 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.428617    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:22.712330 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.712253    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"]
Oct 30 16:53:22.716638 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:53:22.716608    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae7a56f_53f0_42f1_8430_ab5572dbeb4c.slice/crio-e7c799f5b649bfe5b023815aed7b72d568058e57e09956c9c62f738f891f3457 WatchSource:0}: Error finding container e7c799f5b649bfe5b023815aed7b72d568058e57e09956c9c62f738f891f3457: Status 404 returned error can't find the container with id e7c799f5b649bfe5b023815aed7b72d568058e57e09956c9c62f738f891f3457
Oct 30 16:53:22.799108 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:22.799075    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"]
Oct 30 16:53:22.801825 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:53:22.801793    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a7a8c0_b7de_45c2_a3c7_289cf54f2a76.slice/crio-4ec078a7bff46810f8056489ffe31d2307ed1e1613d28e61d810354a121395b2 WatchSource:0}: Error finding container 4ec078a7bff46810f8056489ffe31d2307ed1e1613d28e61d810354a121395b2: Status 404 returned error can't find the container with id 4ec078a7bff46810f8056489ffe31d2307ed1e1613d28e61d810354a121395b2
Oct 30 16:53:23.205724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.205681    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 16:53:23.206161 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.205745    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 16:53:23.676631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.675413    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt" event={"ID":"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76","Type":"ContainerStarted","Data":"da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2"}
Oct 30 16:53:23.676631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.675480    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt" event={"ID":"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76","Type":"ContainerStarted","Data":"4ec078a7bff46810f8056489ffe31d2307ed1e1613d28e61d810354a121395b2"}
Oct 30 16:53:23.676631 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.675897    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:23.678588 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.678540    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq" event={"ID":"eae7a56f-53f0-42f1-8430-ab5572dbeb4c","Type":"ContainerStarted","Data":"b9b434f24a47092c6206e900b04fc001dcffc48fc3efd55eb9dbbef2f66551bb"}
Oct 30 16:53:23.678744 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.678616    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq" event={"ID":"eae7a56f-53f0-42f1-8430-ab5572dbeb4c","Type":"ContainerStarted","Data":"e7c799f5b649bfe5b023815aed7b72d568058e57e09956c9c62f738f891f3457"}
Oct 30 16:53:23.679968 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.679464    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:23.713248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.713171    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt" podStartSLOduration=1.713156676 podStartE2EDuration="1.713156676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:23.712253613 +0000 UTC m=+1833.566131878" watchObservedRunningTime="2026-10-30 16:53:23.713156676 +0000 UTC m=+1833.567034940"
Oct 30 16:53:23.752725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:23.752659    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq" podStartSLOduration=2.455523826 podStartE2EDuration="2.752638653s" totalImagesPullingTime="297.114827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:23.74944663 +0000 UTC m=+1833.603324896" watchObservedRunningTime="2026-10-30 16:53:23.752638653 +0000 UTC m=+1833.606516920"
Oct 30 16:53:24.210619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:24.210585    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:24.211140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:24.211073    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:24.211931 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:24.211910    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:53:30.842240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:30.837376    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 16:53:34.701240 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:34.700993    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 16:53:39.885028 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:39.884993    4497 generic.go:356] "Generic (PLEG): container finished" podID="999975c1-3ece-490c-ad9f-a039410f320a" containerID="2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7" exitCode=0
Oct 30 16:53:39.885510 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:39.885061    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" event={"ID":"999975c1-3ece-490c-ad9f-a039410f320a","Type":"ContainerDied","Data":"2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7"}
Oct 30 16:53:39.885567 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:39.885552    4497 scope.go:122] "RemoveContainer" containerID="2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7"
Oct 30 16:53:39.887061 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:39.887035    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:39.887130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:39.887098    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:39.887169 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:39.887131    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:40.912959 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:40.912918    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" event={"ID":"999975c1-3ece-490c-ad9f-a039410f320a","Type":"ContainerStarted","Data":"43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b"}
Oct 30 16:53:41.418166 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:41.418128    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"]
Oct 30 16:53:41.452692 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:41.452650    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"]
Oct 30 16:53:41.716949 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:41.716865    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:53:41.717135 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:41.717061    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 16:53:41.717227 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:41.717134    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 16:53:42.716464 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:42.716431    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 16:53:42.716843 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:42.716488    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 16:53:43.716666 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:43.716632    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 16:53:43.717084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:43.716695    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 16:53:44.716952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:44.716919    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 16:53:44.717435 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:44.716971    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 16:53:45.716603 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:45.716565    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 16:53:45.716789 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:45.716623    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 16:53:46.422065 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.421267    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk" podUID="c6cde465-ffd3-4c14-b970-f9d98e344ff9" containerName="payload-processing" containerID="cri-o://fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf" gracePeriod=25
Oct 30 16:53:46.712462 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.712434    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:53:46.721321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.721292    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:53:46.721693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.721675    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:53:46.722436 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.722419    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 16:53:46.770528 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.770497    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6cde465-ffd3-4c14-b970-f9d98e344ff9-kube-api-access-4xrl8\" (UniqueName: \"kubernetes.io/projected/c6cde465-ffd3-4c14-b970-f9d98e344ff9-kube-api-access-4xrl8\") pod \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\" (UID: \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\") "
Oct 30 16:53:46.770696 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.770680    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6cde465-ffd3-4c14-b970-f9d98e344ff9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6cde465-ffd3-4c14-b970-f9d98e344ff9-plugins-config-volume\") pod \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\" (UID: \"c6cde465-ffd3-4c14-b970-f9d98e344ff9\") "
Oct 30 16:53:46.771282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.771079    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6cde465-ffd3-4c14-b970-f9d98e344ff9-plugins-config-volume" pod "c6cde465-ffd3-4c14-b970-f9d98e344ff9" (UID: "c6cde465-ffd3-4c14-b970-f9d98e344ff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:53:46.771282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.771251    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6cde465-ffd3-4c14-b970-f9d98e344ff9-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:46.773020 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.772995    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6cde465-ffd3-4c14-b970-f9d98e344ff9-kube-api-access-4xrl8" pod "c6cde465-ffd3-4c14-b970-f9d98e344ff9" (UID: "c6cde465-ffd3-4c14-b970-f9d98e344ff9"). InnerVolumeSpecName "kube-api-access-4xrl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:53:46.872858 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.872825    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xrl8\" (UniqueName: \"kubernetes.io/projected/c6cde465-ffd3-4c14-b970-f9d98e344ff9-kube-api-access-4xrl8\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:46.921826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.921735    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq" podUID="589db689-6c30-44f3-9c9f-b0c36ef2f063" containerName="payload-pre-processing" containerID="cri-o://4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58" gracePeriod=25
Oct 30 16:53:46.969917 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.969879    4497 generic.go:356] "Generic (PLEG): container finished" podID="c6cde465-ffd3-4c14-b970-f9d98e344ff9" containerID="fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf" exitCode=0
Oct 30 16:53:46.970123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.969964    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk" event={"ID":"c6cde465-ffd3-4c14-b970-f9d98e344ff9","Type":"ContainerDied","Data":"fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf"}
Oct 30 16:53:46.970123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.970050    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk" event={"ID":"c6cde465-ffd3-4c14-b970-f9d98e344ff9","Type":"ContainerDied","Data":"186e7289aa3687cd37e36f39e9b91a792ec134302b5d3d8894a151e60e788f8f"}
Oct 30 16:53:46.970123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.970067    4497 scope.go:122] "RemoveContainer" containerID="fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf"
Oct 30 16:53:46.970329 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:46.969972    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"
Oct 30 16:53:47.028997 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.028962    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"]
Oct 30 16:53:47.034764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.034734    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a8ab1-bc77dbd84-d7gqk"]
Oct 30 16:53:47.036558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.036538    4497 scope.go:122] "RemoveContainer" containerID="fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf"
Oct 30 16:53:47.036892 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:47.036872    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf\": container with ID starting with fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf not found: ID does not exist" containerID="fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf"
Oct 30 16:53:47.036951 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.036901    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf"} err="failed to get container status \"fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf\": rpc error: code = NotFound desc = could not find container \"fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf\": container with ID starting with fe8bce07c2f0979dd66a2e736f4ef350602694be04ffaaeeda9d0d5e1e5cedcf not found: ID does not exist"
Oct 30 16:53:47.225005 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.224975    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:53:47.276668 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.276636    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589db689-6c30-44f3-9c9f-b0c36ef2f063-kube-api-access-fqjqr\" (UniqueName: \"kubernetes.io/projected/589db689-6c30-44f3-9c9f-b0c36ef2f063-kube-api-access-fqjqr\") pod \"589db689-6c30-44f3-9c9f-b0c36ef2f063\" (UID: \"589db689-6c30-44f3-9c9f-b0c36ef2f063\") "
Oct 30 16:53:47.276956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.276938    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589db689-6c30-44f3-9c9f-b0c36ef2f063-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589db689-6c30-44f3-9c9f-b0c36ef2f063-plugins-config-volume\") pod \"589db689-6c30-44f3-9c9f-b0c36ef2f063\" (UID: \"589db689-6c30-44f3-9c9f-b0c36ef2f063\") "
Oct 30 16:53:47.277373 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.277230    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589db689-6c30-44f3-9c9f-b0c36ef2f063-plugins-config-volume" pod "589db689-6c30-44f3-9c9f-b0c36ef2f063" (UID: "589db689-6c30-44f3-9c9f-b0c36ef2f063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:53:47.278834 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.278805    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589db689-6c30-44f3-9c9f-b0c36ef2f063-kube-api-access-fqjqr" pod "589db689-6c30-44f3-9c9f-b0c36ef2f063" (UID: "589db689-6c30-44f3-9c9f-b0c36ef2f063"). InnerVolumeSpecName "kube-api-access-fqjqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:53:47.379377 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.379340    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589db689-6c30-44f3-9c9f-b0c36ef2f063-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:47.379582 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.379422    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqjqr\" (UniqueName: \"kubernetes.io/projected/589db689-6c30-44f3-9c9f-b0c36ef2f063-kube-api-access-fqjqr\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:53:47.977844 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.977807    4497 generic.go:356] "Generic (PLEG): container finished" podID="589db689-6c30-44f3-9c9f-b0c36ef2f063" containerID="4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58" exitCode=0
Oct 30 16:53:47.978346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.977855    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"
Oct 30 16:53:47.978346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.977886    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq" event={"ID":"589db689-6c30-44f3-9c9f-b0c36ef2f063","Type":"ContainerDied","Data":"4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58"}
Oct 30 16:53:47.978346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.977943    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq" event={"ID":"589db689-6c30-44f3-9c9f-b0c36ef2f063","Type":"ContainerDied","Data":"aba513934d49719630a19690cf436e0aa47cf062573f6f6ecf00a01dad153e9f"}
Oct 30 16:53:47.978346 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.977967    4497 scope.go:122] "RemoveContainer" containerID="4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58"
Oct 30 16:53:47.992369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.991699    4497 scope.go:122] "RemoveContainer" containerID="4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58"
Oct 30 16:53:47.992369 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:47.992139    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58\": container with ID starting with 4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58 not found: ID does not exist" containerID="4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58"
Oct 30 16:53:47.992369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:47.992177    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58"} err="failed to get container status \"4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58\": rpc error: code = NotFound desc = could not find container \"4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58\": container with ID starting with 4089f3970bcab3a0ba053a4a3a242e6cf391313c30d730597b6d6c786a9bae58 not found: ID does not exist"
Oct 30 16:53:48.035458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:48.035411    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"]
Oct 30 16:53:48.045273 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:48.045243    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4a8ab1-764db466f5-dbrwq"]
Oct 30 16:53:48.300441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:48.300405    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589db689-6c30-44f3-9c9f-b0c36ef2f063" path="/var/lib/kubelet/pods/589db689-6c30-44f3-9c9f-b0c36ef2f063/volumes"
Oct 30 16:53:48.300891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:48.300874    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6cde465-ffd3-4c14-b970-f9d98e344ff9" path="/var/lib/kubelet/pods/c6cde465-ffd3-4c14-b970-f9d98e344ff9/volumes"
Oct 30 16:53:51.563129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.562386    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"]
Oct 30 16:53:51.563129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.563094    4497 cpu_manager.go:414] "containerMap: removing container" podUID="589db689-6c30-44f3-9c9f-b0c36ef2f063" containerName="payload-pre-processing"
Oct 30 16:53:51.563129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.563112    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589db689-6c30-44f3-9c9f-b0c36ef2f063" containerName="payload-pre-processing"
Oct 30 16:53:51.563743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.563145    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c6cde465-ffd3-4c14-b970-f9d98e344ff9" containerName="payload-processing"
Oct 30 16:53:51.563743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.563151    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6cde465-ffd3-4c14-b970-f9d98e344ff9" containerName="payload-processing"
Oct 30 16:53:51.563743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.563273    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="589db689-6c30-44f3-9c9f-b0c36ef2f063" containerName="payload-pre-processing"
Oct 30 16:53:51.563743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.563288    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6cde465-ffd3-4c14-b970-f9d98e344ff9" containerName="payload-processing"
Oct 30 16:53:51.569308 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.568656    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.581139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.581107    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8bff6e-openshift-default-dockercfg-qz7cj\""
Oct 30 16:53:51.595672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.595637    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"]
Oct 30 16:53:51.656957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.656915    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-credential-socket\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.657159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.656977    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-socket\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.657159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.657038    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-envoy\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.657312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.657179    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff542801-6f7a-4588-9f8b-dac74cae73eb-istiod-ca-cert\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.657312 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.657304    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-data\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.657389 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.657352    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-token\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.657389 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.657372    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxhp\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-kube-api-access-8vxhp\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.657467 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.657406    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-certs\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.657607 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.657587    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-podinfo\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765154    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-data\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765258    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-token\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765300    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxhp\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-kube-api-access-8vxhp\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765344    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-certs\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765516    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-podinfo\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765579    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-credential-socket\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765629    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-socket\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765651    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-envoy\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.765732    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff542801-6f7a-4588-9f8b-dac74cae73eb-istiod-ca-cert\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.766835 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.766668    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff542801-6f7a-4588-9f8b-dac74cae73eb-istiod-ca-cert\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.773212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.767015    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-data\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.773212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.767783    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-certs\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.773212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.768536    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-socket\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.773212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.768772    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-credential-socket\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.815925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.800725    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-token\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.815925 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.812053    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxhp\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-kube-api-access-8vxhp\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.825212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.816917    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-envoy\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.825212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.817525    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-podinfo\") pod \"e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") " pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.890765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:51.890733    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:51.930852 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:53:51.930812    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:53:52.243957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:52.243613    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"]
Oct 30 16:53:52.246892 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:53:52.246861    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff542801_6f7a_4588_9f8b_dac74cae73eb.slice/crio-cf79d01a6b97a53e410ffa94cc1a5f3e02c17670710918aaf693d6a7ccc3ae4b WatchSource:0}: Error finding container cf79d01a6b97a53e410ffa94cc1a5f3e02c17670710918aaf693d6a7ccc3ae4b: Status 404 returned error can't find the container with id cf79d01a6b97a53e410ffa94cc1a5f3e02c17670710918aaf693d6a7ccc3ae4b
Oct 30 16:53:52.249590 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:52.249553    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:52.249680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:52.249626    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:52.249680 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:52.249657    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:53:53.020130 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:53.020087    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" event={"ID":"ff542801-6f7a-4588-9f8b-dac74cae73eb","Type":"ContainerStarted","Data":"5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7"}
Oct 30 16:53:53.020591 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:53.020149    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" event={"ID":"ff542801-6f7a-4588-9f8b-dac74cae73eb","Type":"ContainerStarted","Data":"cf79d01a6b97a53e410ffa94cc1a5f3e02c17670710918aaf693d6a7ccc3ae4b"}
Oct 30 16:53:53.079162 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:53.079109    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podStartSLOduration=2.079095071 podStartE2EDuration="2.079095071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:53.069518871 +0000 UTC m=+1862.923397138" watchObservedRunningTime="2026-10-30 16:53:53.079095071 +0000 UTC m=+1862.932973335"
Oct 30 16:53:53.891349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:53.891320    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:53.897179 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:53.897146    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:54.104210 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:54.102539    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:54.109260 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:54.108405    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:53:57.401107 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.400254    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"]
Oct 30 16:53:57.406072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.406049    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:57.430715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.430685    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8bff6e\""
Oct 30 16:53:57.434032 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.433999    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"]
Oct 30 16:53:57.452381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.452348    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sq58\" (UniqueName: \"kubernetes.io/projected/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-kube-api-access-5sq58\") pod \"payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg\" (UID: \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:57.452532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.452514    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg\" (UID: \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:57.464657 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.464627    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"]
Oct 30 16:53:57.468763 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.468746    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:57.500212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.488504    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8bff6e-dockercfg-f8jwl\""
Oct 30 16:53:57.504394 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.504368    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"]
Oct 30 16:53:57.553633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.553599    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr\" (UID: \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:57.554001 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.553977    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg\" (UID: \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:57.554165 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.554029    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h565w\" (UniqueName: \"kubernetes.io/projected/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-kube-api-access-h565w\") pod \"payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr\" (UID: \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:57.554377 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.554256    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sq58\" (UniqueName: \"kubernetes.io/projected/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-kube-api-access-5sq58\") pod \"payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg\" (UID: \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:57.555216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.555181    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg\" (UID: \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:57.639216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.638985    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sq58\" (UniqueName: \"kubernetes.io/projected/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-kube-api-access-5sq58\") pod \"payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg\" (UID: \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:57.664315 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.658582    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr\" (UID: \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:57.664315 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.658747    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h565w\" (UniqueName: \"kubernetes.io/projected/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-kube-api-access-h565w\") pod \"payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr\" (UID: \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:57.691231 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.688291    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr\" (UID: \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:57.725026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.724393    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h565w\" (UniqueName: \"kubernetes.io/projected/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-kube-api-access-h565w\") pod \"payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr\" (UID: \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:57.725026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.724653    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:57.832460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:57.832017    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:58.098805 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:58.098772    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"]
Oct 30 16:53:58.101289 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:53:58.101265    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90987f2_2ff8_49cb_a3a9_d4e398ee1928.slice/crio-305e81e34de0e1fd4cf8c998022f8136b7b7e98ad496e76a0d07086f11f5e8b1 WatchSource:0}: Error finding container 305e81e34de0e1fd4cf8c998022f8136b7b7e98ad496e76a0d07086f11f5e8b1: Status 404 returned error can't find the container with id 305e81e34de0e1fd4cf8c998022f8136b7b7e98ad496e76a0d07086f11f5e8b1
Oct 30 16:53:58.126352 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:58.126323    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg" event={"ID":"e90987f2-2ff8-49cb-a3a9-d4e398ee1928","Type":"ContainerStarted","Data":"305e81e34de0e1fd4cf8c998022f8136b7b7e98ad496e76a0d07086f11f5e8b1"}
Oct 30 16:53:58.200137 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:58.199865    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"]
Oct 30 16:53:58.200137 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:53:58.199878    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fde66f_17f2_4ccf_bf68_c5f5b6cdfb59.slice/crio-7751f8bc0d16931319b6e70fea283b19b4282094c0f766363d896aa5722ef446 WatchSource:0}: Error finding container 7751f8bc0d16931319b6e70fea283b19b4282094c0f766363d896aa5722ef446: Status 404 returned error can't find the container with id 7751f8bc0d16931319b6e70fea283b19b4282094c0f766363d896aa5722ef446
Oct 30 16:53:59.134733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:59.134694    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg" event={"ID":"e90987f2-2ff8-49cb-a3a9-d4e398ee1928","Type":"ContainerStarted","Data":"87866cfcabbc8cfd783617bb3d0a6277595a73248596eae14bb09d25d63481a5"}
Oct 30 16:53:59.135141 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:59.134762    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:53:59.136356 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:59.136330    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr" event={"ID":"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59","Type":"ContainerStarted","Data":"c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093"}
Oct 30 16:53:59.136485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:59.136370    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr" event={"ID":"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59","Type":"ContainerStarted","Data":"7751f8bc0d16931319b6e70fea283b19b4282094c0f766363d896aa5722ef446"}
Oct 30 16:53:59.136485 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:59.136416    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:53:59.193212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:59.188273    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg" podStartSLOduration=2.188251173 podStartE2EDuration="2.188251173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:59.170835744 +0000 UTC m=+1869.024714002" watchObservedRunningTime="2026-10-30 16:53:59.188251173 +0000 UTC m=+1869.042129439"
Oct 30 16:53:59.216800 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:53:59.216736    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr" podStartSLOduration=2.216721137 podStartE2EDuration="2.216721137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:59.212328993 +0000 UTC m=+1869.066207262" watchObservedRunningTime="2026-10-30 16:53:59.216721137 +0000 UTC m=+1869.070599401"
Oct 30 16:54:01.213431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.213398    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"]
Oct 30 16:54:01.213924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.213672    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw" podUID="71b2692b-f6c7-47e0-88ce-7c57c9213bfe" containerName="maas-api" containerID="cri-o://38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf" gracePeriod=30
Oct 30 16:54:01.535868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.535518    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:54:01.655336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.655307    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-kube-api-access-gjc89\" (UniqueName: \"kubernetes.io/projected/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-kube-api-access-gjc89\") pod \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") "
Oct 30 16:54:01.655528 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.655348    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-openshift-service-ca\") pod \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") "
Oct 30 16:54:01.655528 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.655436    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-maas-api-tls\") pod \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") "
Oct 30 16:54:01.655528 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.655465    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-metrics-certs\") pod \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\" (UID: \"71b2692b-f6c7-47e0-88ce-7c57c9213bfe\") "
Oct 30 16:54:01.655816 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.655780    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-openshift-service-ca" pod "71b2692b-f6c7-47e0-88ce-7c57c9213bfe" (UID: "71b2692b-f6c7-47e0-88ce-7c57c9213bfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:54:01.656448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.656419    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:01.657750 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.657724    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-metrics-certs" pod "71b2692b-f6c7-47e0-88ce-7c57c9213bfe" (UID: "71b2692b-f6c7-47e0-88ce-7c57c9213bfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:54:01.657864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.657788    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-maas-api-tls" pod "71b2692b-f6c7-47e0-88ce-7c57c9213bfe" (UID: "71b2692b-f6c7-47e0-88ce-7c57c9213bfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:54:01.657864 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.657797    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-kube-api-access-gjc89" pod "71b2692b-f6c7-47e0-88ce-7c57c9213bfe" (UID: "71b2692b-f6c7-47e0-88ce-7c57c9213bfe"). InnerVolumeSpecName "kube-api-access-gjc89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:54:01.760023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.759987    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjc89\" (UniqueName: \"kubernetes.io/projected/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-kube-api-access-gjc89\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:01.760023 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.760023    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:01.760334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:01.760035    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71b2692b-f6c7-47e0-88ce-7c57c9213bfe-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:02.158348 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.158251    4497 generic.go:356] "Generic (PLEG): container finished" podID="71b2692b-f6c7-47e0-88ce-7c57c9213bfe" containerID="38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf" exitCode=0
Oct 30 16:54:02.158348 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.158314    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"
Oct 30 16:54:02.158558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.158338    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw" event={"ID":"71b2692b-f6c7-47e0-88ce-7c57c9213bfe","Type":"ContainerDied","Data":"38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf"}
Oct 30 16:54:02.158558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.158431    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw" event={"ID":"71b2692b-f6c7-47e0-88ce-7c57c9213bfe","Type":"ContainerDied","Data":"a4c2da1e8ba450634c5032d4466594ad1f9c4656644da3d21786bb5f107d747d"}
Oct 30 16:54:02.158558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.158463    4497 scope.go:122] "RemoveContainer" containerID="38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf"
Oct 30 16:54:02.171125 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.171098    4497 scope.go:122] "RemoveContainer" containerID="38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf"
Oct 30 16:54:02.171586 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:54:02.171548    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf\": container with ID starting with 38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf not found: ID does not exist" containerID="38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf"
Oct 30 16:54:02.171693 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.171600    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf"} err="failed to get container status \"38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf\": rpc error: code = NotFound desc = could not find container \"38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf\": container with ID starting with 38ab3de936c36205c309eb0038033061d9191fe01ea1306569132316eef8efaf not found: ID does not exist"
Oct 30 16:54:02.205418 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.205384    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"]
Oct 30 16:54:02.214414 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.214371    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4a8ab1-769bf7f858-7jqbw"]
Oct 30 16:54:02.302822 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:02.302784    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b2692b-f6c7-47e0-88ce-7c57c9213bfe" path="/var/lib/kubelet/pods/71b2692b-f6c7-47e0-88ce-7c57c9213bfe/volumes"
Oct 30 16:54:05.183120 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:05.183084    4497 generic.go:356] "Generic (PLEG): container finished" podID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerID="f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b" exitCode=0
Oct 30 16:54:05.183585 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:05.183158    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" event={"ID":"0a0fca89-11f6-4450-9b7b-27f9fe014668","Type":"ContainerDied","Data":"f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b"}
Oct 30 16:54:05.183665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:05.183651    4497 scope.go:122] "RemoveContainer" containerID="f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b"
Oct 30 16:54:05.185602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:05.185567    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:54:05.185682 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:05.185664    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:54:05.185736 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:05.185720    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:54:06.202840 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:06.202803    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" event={"ID":"0a0fca89-11f6-4450-9b7b-27f9fe014668","Type":"ContainerStarted","Data":"92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a"}
Oct 30 16:54:06.566241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:06.566182    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:54:06.566421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:06.566380    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 16:54:06.566465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:06.566428    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 16:54:07.565910 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:07.565874    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 16:54:07.566347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:07.565931    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 16:54:08.565860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:08.565822    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 16:54:08.566048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:08.565904    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 16:54:09.566144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:09.566105    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 16:54:09.566600 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:09.566160    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 16:54:10.145948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:10.145914    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 16:54:10.146134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:10.145993    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 16:54:10.566222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:10.566168    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 16:54:10.566707 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:10.566245    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 16:54:11.569923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:11.569890    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:54:11.570336 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:11.570215    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:54:11.570894 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:11.570877    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 16:54:14.219412 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.219378    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"]
Oct 30 16:54:14.220016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.220001    4497 cpu_manager.go:414] "containerMap: removing container" podUID="71b2692b-f6c7-47e0-88ce-7c57c9213bfe" containerName="maas-api"
Oct 30 16:54:14.220062 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.220018    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b2692b-f6c7-47e0-88ce-7c57c9213bfe" containerName="maas-api"
Oct 30 16:54:14.220127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.220118    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b2692b-f6c7-47e0-88ce-7c57c9213bfe" containerName="maas-api"
Oct 30 16:54:14.223566 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.223550    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.235478 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.235459    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-deb721-serving-cert\""
Oct 30 16:54:14.249136 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.249112    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"]
Oct 30 16:54:14.290144 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.290114    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.290335 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.290166    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.290377 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.290357    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj97g\" (UniqueName: \"kubernetes.io/projected/2106064d-663d-4cfa-9c7d-84eca30a97e1-kube-api-access-cj97g\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.290481 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.290465    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2106064d-663d-4cfa-9c7d-84eca30a97e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.391888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.391849    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.392097 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.391906    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.392097 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.391988    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj97g\" (UniqueName: \"kubernetes.io/projected/2106064d-663d-4cfa-9c7d-84eca30a97e1-kube-api-access-cj97g\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.392097 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.392069    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2106064d-663d-4cfa-9c7d-84eca30a97e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.392764 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.392738    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2106064d-663d-4cfa-9c7d-84eca30a97e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.394443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.394423    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.394517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.394461    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.412573 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.412547    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj97g\" (UniqueName: \"kubernetes.io/projected/2106064d-663d-4cfa-9c7d-84eca30a97e1-kube-api-access-cj97g\") pod \"maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.536808 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.536774    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:14.860114 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:14.860089    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"]
Oct 30 16:54:14.863233 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:54:14.863207    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2106064d_663d_4cfa_9c7d_84eca30a97e1.slice/crio-c98051db0b70ded644e5a511ceff9ea921f7ef5937f2fd551319cb7d8b8d21cc WatchSource:0}: Error finding container c98051db0b70ded644e5a511ceff9ea921f7ef5937f2fd551319cb7d8b8d21cc: Status 404 returned error can't find the container with id c98051db0b70ded644e5a511ceff9ea921f7ef5937f2fd551319cb7d8b8d21cc
Oct 30 16:54:15.287155 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:15.287067    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s" event={"ID":"2106064d-663d-4cfa-9c7d-84eca30a97e1","Type":"ContainerStarted","Data":"c98051db0b70ded644e5a511ceff9ea921f7ef5937f2fd551319cb7d8b8d21cc"}
Oct 30 16:54:16.303754 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:16.303716    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:16.304158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:16.303790    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s" event={"ID":"2106064d-663d-4cfa-9c7d-84eca30a97e1","Type":"ContainerStarted","Data":"a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55"}
Oct 30 16:54:16.380946 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:16.380880    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s" podStartSLOduration=2.003469014 podStartE2EDuration="2.380858192s" totalImagesPullingTime="377.389178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:16.379876866 +0000 UTC m=+1886.233755137" watchObservedRunningTime="2026-10-30 16:54:16.380858192 +0000 UTC m=+1886.234736460"
Oct 30 16:54:19.267847 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.267805    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"]
Oct 30 16:54:19.273357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.273327    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.293501 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.293470    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"]
Oct 30 16:54:19.296366 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.296345    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-799c76-openshift-default-dockercfg-l4jm5\""
Oct 30 16:54:19.381022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.380978    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-certs\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.381249 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.381079    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-token\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.381339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.381247    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-credential-socket\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.381339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.381289    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4242887-cbd0-4d9a-ae41-732a0eec8669-istiod-ca-cert\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.381339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.381335    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-socket\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.381339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.381365    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-podinfo\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.381597 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.381395    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddcw\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-kube-api-access-lddcw\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.381597 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.381447    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-envoy\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.381700 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.381673    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-data\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.494294 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.494252    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-data\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.494583 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.494566    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-certs\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.494745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.494732    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-token\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.494914 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.494902    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-credential-socket\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.495018 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.495006    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4242887-cbd0-4d9a-ae41-732a0eec8669-istiod-ca-cert\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.495122 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.495109    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-socket\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.495232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.495218    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-podinfo\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.495344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.495330    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddcw\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-kube-api-access-lddcw\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.495451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.495440    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-envoy\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.496344 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.496319    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-credential-socket\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.507675 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.507641    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-certs\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.507902 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.496681    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-socket\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.508293 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.508272    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-data\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.524296 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.508844    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4242887-cbd0-4d9a-ae41-732a0eec8669-istiod-ca-cert\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.524785 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.524757    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-envoy\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.550448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.549908    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddcw\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-kube-api-access-lddcw\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.550448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.550396    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-podinfo\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.551125 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.551103    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-token\") pod \"e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") " pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.592392 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.591906    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:19.940328 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.940251    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"]
Oct 30 16:54:19.944131 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:54:19.944102    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4242887_cbd0_4d9a_ae41_732a0eec8669.slice/crio-812f5b6c22dd4d320bbf5e32bb745749408abdb0a509e7fbf2e58a4119bc897b WatchSource:0}: Error finding container 812f5b6c22dd4d320bbf5e32bb745749408abdb0a509e7fbf2e58a4119bc897b: Status 404 returned error can't find the container with id 812f5b6c22dd4d320bbf5e32bb745749408abdb0a509e7fbf2e58a4119bc897b
Oct 30 16:54:19.946860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.946823    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:54:19.946932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.946895    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:54:19.946932 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:19.946926    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:54:20.331433 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:20.331392    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" event={"ID":"d4242887-cbd0-4d9a-ae41-732a0eec8669","Type":"ContainerStarted","Data":"66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302"}
Oct 30 16:54:20.332052 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:20.331458    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" event={"ID":"d4242887-cbd0-4d9a-ae41-732a0eec8669","Type":"ContainerStarted","Data":"812f5b6c22dd4d320bbf5e32bb745749408abdb0a509e7fbf2e58a4119bc897b"}
Oct 30 16:54:20.383245 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:20.383173    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" podStartSLOduration=1.38315833 podStartE2EDuration="1.38315833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:20.381544786 +0000 UTC m=+1890.235423050" watchObservedRunningTime="2026-10-30 16:54:20.38315833 +0000 UTC m=+1890.237036599"
Oct 30 16:54:20.593356 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:20.593275    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:21.605383 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:21.605349    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:21.921901 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:54:21.919669    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:54:22.312642 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:22.312610    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 16:54:22.351753 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:22.351725    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:22.353381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:22.353353    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:54:35.856140 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:35.856104    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"]
Oct 30 16:54:35.856575 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:35.856337    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy" containerID="cri-o://1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71" gracePeriod=30
Oct 30 16:54:41.152339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.152312    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:54:41.339171 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.339131    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-envoy\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.339353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.339214    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-data\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.339353 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.339342    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-certs\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.339525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.339385    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-token\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-token\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.339525 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.339508    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-podinfo\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.339637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.339535    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444f234d-59b5-4fc5-b736-2fa11c30aa91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444f234d-59b5-4fc5-b736-2fa11c30aa91-istiod-ca-cert\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.339707 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.339635    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-data" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:54:41.339707 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.339672    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-kube-api-access-x9tq7\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-kube-api-access-x9tq7\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.340960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.340344    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-socket\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.340960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.340395    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444f234d-59b5-4fc5-b736-2fa11c30aa91-istiod-ca-cert" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:54:41.340960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.340418    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-credential-socket\") pod \"444f234d-59b5-4fc5-b736-2fa11c30aa91\" (UID: \"444f234d-59b5-4fc5-b736-2fa11c30aa91\") "
Oct 30 16:54:41.340960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.340614    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-socket" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:54:41.340960 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.340900    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-credential-socket" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:54:41.341289 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.340965    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-certs" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:54:41.341547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.341354    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.341547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.341382    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.341547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.341400    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.341547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.341416    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.341547 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.341430    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444f234d-59b5-4fc5-b736-2fa11c30aa91-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.342297 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.342274    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-token" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:54:41.342418 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.342315    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-kube-api-access-x9tq7" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "kube-api-access-x9tq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:54:41.342613 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.342595    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-envoy" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:54:41.342885 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.342858    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-podinfo" pod "444f234d-59b5-4fc5-b736-2fa11c30aa91" (UID: "444f234d-59b5-4fc5-b736-2fa11c30aa91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:54:41.443342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.443256    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.443342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.443291    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9tq7\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-kube-api-access-x9tq7\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.443342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.443306    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.443342 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.443319    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/444f234d-59b5-4fc5-b736-2fa11c30aa91-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:54:41.476780 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.476745    4497 generic.go:356] "Generic (PLEG): container finished" podID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerID="1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71" exitCode=0
Oct 30 16:54:41.476964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.476793    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"
Oct 30 16:54:41.476964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.476813    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" event={"ID":"444f234d-59b5-4fc5-b736-2fa11c30aa91","Type":"ContainerDied","Data":"1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71"}
Oct 30 16:54:41.476964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.476866    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq" event={"ID":"444f234d-59b5-4fc5-b736-2fa11c30aa91","Type":"ContainerDied","Data":"2626f9f0b42a26436ad6f66ab893ab0b4a26b46b28dc9b8ec804be85f6cd228c"}
Oct 30 16:54:41.476964 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.476895    4497 scope.go:122] "RemoveContainer" containerID="1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71"
Oct 30 16:54:41.487521 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.487500    4497 scope.go:122] "RemoveContainer" containerID="1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71"
Oct 30 16:54:41.487801 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:54:41.487778    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71\": container with ID starting with 1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71 not found: ID does not exist" containerID="1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71"
Oct 30 16:54:41.487850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.487804    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71"} err="failed to get container status \"1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71\": rpc error: code = NotFound desc = could not find container \"1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71\": container with ID starting with 1b9c080162bf785dddaa9e546c3b786136ce892da082c2adf0cbf897d1405d71 not found: ID does not exist"
Oct 30 16:54:41.522149 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.522118    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"]
Oct 30 16:54:41.531570 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:41.531546    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wn9fq"]
Oct 30 16:54:42.062999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.062968    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"]
Oct 30 16:54:42.063676 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.063661    4497 cpu_manager.go:414] "containerMap: removing container" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy"
Oct 30 16:54:42.063737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.063678    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy"
Oct 30 16:54:42.063785 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.063777    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" containerName="istio-proxy"
Oct 30 16:54:42.073880 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.073853    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.088508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.088470    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"]
Oct 30 16:54:42.093903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.093880    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-3ca339-serving-cert\""
Oct 30 16:54:42.255349 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.255315    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfj5\" (UniqueName: \"kubernetes.io/projected/fa4050d9-892d-47e9-adf4-96a72db5b1a5-kube-api-access-bkfj5\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.255818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.255359    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.255818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.255501    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.255818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.255556    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4050d9-892d-47e9-adf4-96a72db5b1a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.301141 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.301100    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444f234d-59b5-4fc5-b736-2fa11c30aa91" path="/var/lib/kubelet/pods/444f234d-59b5-4fc5-b736-2fa11c30aa91/volumes"
Oct 30 16:54:42.370026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.368842    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfj5\" (UniqueName: \"kubernetes.io/projected/fa4050d9-892d-47e9-adf4-96a72db5b1a5-kube-api-access-bkfj5\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.370026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.368908    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.370026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.369017    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.370026 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.369082    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4050d9-892d-47e9-adf4-96a72db5b1a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.370415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.370358    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4050d9-892d-47e9-adf4-96a72db5b1a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.373594 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.373568    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.373746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.373692    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.392299 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.392265    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfj5\" (UniqueName: \"kubernetes.io/projected/fa4050d9-892d-47e9-adf4-96a72db5b1a5-kube-api-access-bkfj5\") pod \"maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.688641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.688545    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:42.999745 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:42.999674    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"]
Oct 30 16:54:43.002401 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:54:43.002375    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4050d9_892d_47e9_adf4_96a72db5b1a5.slice/crio-2574f3f7ef01f22b8beb52dbf2339a230e3f99bea91bcef13b7f14650cfe24a1 WatchSource:0}: Error finding container 2574f3f7ef01f22b8beb52dbf2339a230e3f99bea91bcef13b7f14650cfe24a1: Status 404 returned error can't find the container with id 2574f3f7ef01f22b8beb52dbf2339a230e3f99bea91bcef13b7f14650cfe24a1
Oct 30 16:54:43.492650 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:43.492612    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs" event={"ID":"fa4050d9-892d-47e9-adf4-96a72db5b1a5","Type":"ContainerStarted","Data":"aba7d70f34fd8b10f906d1e3a247042a6979d08b086f44108d7d27fbe9a49c0c"}
Oct 30 16:54:43.493164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:43.492693    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs" event={"ID":"fa4050d9-892d-47e9-adf4-96a72db5b1a5","Type":"ContainerStarted","Data":"2574f3f7ef01f22b8beb52dbf2339a230e3f99bea91bcef13b7f14650cfe24a1"}
Oct 30 16:54:43.493733 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:43.493707    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:43.531374 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:43.531304    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs" podStartSLOduration=1.201638658 podStartE2EDuration="1.531290379s" totalImagesPullingTime="329.651721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:43.531037382 +0000 UTC m=+1913.384915647" watchObservedRunningTime="2026-10-30 16:54:43.531290379 +0000 UTC m=+1913.385168652"
Oct 30 16:54:50.102404 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.102365    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"]
Oct 30 16:54:50.107007 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.106977    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.119116 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.119094    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-799c76-serving-cert\""
Oct 30 16:54:50.133254 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.133225    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"]
Oct 30 16:54:50.203865 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.203822    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzstm\" (UniqueName: \"kubernetes.io/projected/f4de5d87-7ec5-46ee-ba50-1fc96353620d-kube-api-access-jzstm\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.204087 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.204066    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de5d87-7ec5-46ee-ba50-1fc96353620d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.204206 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.204161    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.204298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.204226    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.305396 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.305360    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de5d87-7ec5-46ee-ba50-1fc96353620d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.305565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.305452    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.305565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.305476    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.305565 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.305526    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzstm\" (UniqueName: \"kubernetes.io/projected/f4de5d87-7ec5-46ee-ba50-1fc96353620d-kube-api-access-jzstm\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.306038 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.306016    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de5d87-7ec5-46ee-ba50-1fc96353620d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.307921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.307900    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.316952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.316931    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-799c76-serving-cert\""
Oct 30 16:54:50.327990 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.327960    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.328298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.328276    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzstm\" (UniqueName: \"kubernetes.io/projected/f4de5d87-7ec5-46ee-ba50-1fc96353620d-kube-api-access-jzstm\") pod \"maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.420776 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.420691    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:50.520890 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.520859    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 16:54:50.746310 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:50.746145    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"]
Oct 30 16:54:50.749309 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:54:50.749280    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4de5d87_7ec5_46ee_ba50_1fc96353620d.slice/crio-ed19e6370bf411484e3605a50723273e5722de1df9b55e117bfbb289aa1e84ed WatchSource:0}: Error finding container ed19e6370bf411484e3605a50723273e5722de1df9b55e117bfbb289aa1e84ed: Status 404 returned error can't find the container with id ed19e6370bf411484e3605a50723273e5722de1df9b55e117bfbb289aa1e84ed
Oct 30 16:54:51.558093 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:51.558006    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq" event={"ID":"f4de5d87-7ec5-46ee-ba50-1fc96353620d","Type":"ContainerStarted","Data":"6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f"}
Oct 30 16:54:51.558093 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:51.558065    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq" event={"ID":"f4de5d87-7ec5-46ee-ba50-1fc96353620d","Type":"ContainerStarted","Data":"ed19e6370bf411484e3605a50723273e5722de1df9b55e117bfbb289aa1e84ed"}
Oct 30 16:54:51.558093 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:51.558100    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:54:51.608579 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:51.608518    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq" podStartSLOduration=1.26078691 podStartE2EDuration="1.608496926s" totalImagesPullingTime="347.710016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:51.596826353 +0000 UTC m=+1921.450704626" watchObservedRunningTime="2026-10-30 16:54:51.608496926 +0000 UTC m=+1921.462375192"
Oct 30 16:54:51.934725 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:54:51.934643    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:54:57.571762 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:54:57.571726    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 16:55:00.683972 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:00.683939    4497 generic.go:356] "Generic (PLEG): container finished" podID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerID="5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7" exitCode=0
Oct 30 16:55:00.684441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:00.684007    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" event={"ID":"ff542801-6f7a-4588-9f8b-dac74cae73eb","Type":"ContainerDied","Data":"5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7"}
Oct 30 16:55:00.684516 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:00.684501    4497 scope.go:122] "RemoveContainer" containerID="5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7"
Oct 30 16:55:00.686164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:00.686133    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:55:00.686271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:00.686220    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:55:00.686271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:00.686256    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:55:01.705217 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:01.705169    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" event={"ID":"ff542801-6f7a-4588-9f8b-dac74cae73eb","Type":"ContainerStarted","Data":"4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3"}
Oct 30 16:55:01.894211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:01.892584    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:55:01.894211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:01.892709    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 16:55:01.894211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:01.892749    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 16:55:02.893271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:02.892255    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 16:55:02.893271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:02.892313    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 16:55:03.896340 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:03.896299    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 16:55:03.896906 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:03.896362    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 16:55:04.896641 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:04.896606    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 16:55:04.897073 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:04.896670    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 16:55:05.896256 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:05.894585    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 16:55:05.896256 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:05.894652    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 16:55:06.897488 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:06.897351    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 16:55:06.897488 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:06.897428    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 16:55:07.896141 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:07.896113    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:55:07.896550 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:07.896530    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:55:07.897155 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:07.897134    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 16:55:21.916437 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:55:21.916399    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:55:22.205426 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.205335    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"]
Oct 30 16:55:22.210943 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.210913    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.226369 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.226337    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"]
Oct 30 16:55:22.231345 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.231316    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-deb721\"/\"openshift-service-ca.crt\""
Oct 30 16:55:22.231531 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.231373    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-deb721\"/\"kube-root-ca.crt\""
Oct 30 16:55:22.232401 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.232378    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-deb721\"/\"auth-test-model-deb721-kserve-self-signed-certs\""
Oct 30 16:55:22.232496 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.232422    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-deb721\"/\"default-dockercfg-p4zj7\""
Oct 30 16:55:22.296134 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.296106    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-model-cache\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.296350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.296246    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-dshm\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.296350 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.296309    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tmp-dir\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.296447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.296427    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-home\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.296500 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.296487    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k747\" (UniqueName: \"kubernetes.io/projected/4b94b372-99c7-4b87-9ecf-f54e87ee055c-kube-api-access-6k747\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.296592 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.296577    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tls-certs\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.397698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.397663    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-model-cache\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.397887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.397725    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-dshm\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.397887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.397763    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tmp-dir\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.397887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.397838    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-home\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.398123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.397895    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k747\" (UniqueName: \"kubernetes.io/projected/4b94b372-99c7-4b87-9ecf-f54e87ee055c-kube-api-access-6k747\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.398123 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.397938    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tls-certs\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.398286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.398122    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-model-cache\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.398286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.398264    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tmp-dir\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.398379 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.398350    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-home\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.400154 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.400131    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-dshm\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.400507 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.400491    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tls-certs\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.418517 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.418488    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k747\" (UniqueName: \"kubernetes.io/projected/4b94b372-99c7-4b87-9ecf-f54e87ee055c-kube-api-access-6k747\") pod \"auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") " pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.531880 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.531843    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:22.851339 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:22.851259    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"]
Oct 30 16:55:22.854598 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:55:22.854553    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b94b372_99c7_4b87_9ecf_f54e87ee055c.slice/crio-9db57768bc4b097924b6961c26c00cf46b1a5604f7dce9c638ab10bf1a53ffd4 WatchSource:0}: Error finding container 9db57768bc4b097924b6961c26c00cf46b1a5604f7dce9c638ab10bf1a53ffd4: Status 404 returned error can't find the container with id 9db57768bc4b097924b6961c26c00cf46b1a5604f7dce9c638ab10bf1a53ffd4
Oct 30 16:55:23.876949 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:23.876898    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl" event={"ID":"4b94b372-99c7-4b87-9ecf-f54e87ee055c","Type":"ContainerStarted","Data":"fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e"}
Oct 30 16:55:23.877427 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:23.876966    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl" event={"ID":"4b94b372-99c7-4b87-9ecf-f54e87ee055c","Type":"ContainerStarted","Data":"9db57768bc4b097924b6961c26c00cf46b1a5604f7dce9c638ab10bf1a53ffd4"}
Oct 30 16:55:23.935923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:23.935871    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl" podStartSLOduration=1.935856189 podStartE2EDuration="1.935856189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:23.934321717 +0000 UTC m=+1953.788199981" watchObservedRunningTime="2026-10-30 16:55:23.935856189 +0000 UTC m=+1953.789734497"
Oct 30 16:55:24.531973 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:24.531930    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:32.532708 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:32.532664    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:32.547410 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:32.547380    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:33.069505 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:33.069476    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:55:35.073921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:35.073890    4497 generic.go:356] "Generic (PLEG): container finished" podID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerID="66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302" exitCode=0
Oct 30 16:55:35.074377 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:35.073963    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" event={"ID":"d4242887-cbd0-4d9a-ae41-732a0eec8669","Type":"ContainerDied","Data":"66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302"}
Oct 30 16:55:35.074441 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:35.074427    4497 scope.go:122] "RemoveContainer" containerID="66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302"
Oct 30 16:55:35.076081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:35.076055    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:55:35.076150 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:35.076121    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:55:35.076208 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:35.076157    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:55:36.104334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:36.104301    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" event={"ID":"d4242887-cbd0-4d9a-ae41-732a0eec8669","Type":"ContainerStarted","Data":"d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8"}
Oct 30 16:55:36.596413 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:36.596377    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:55:36.597602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:36.597572    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 16:55:36.597729 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:36.597619    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 16:55:37.352000 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:37.351966    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:55:37.595645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:37.595602    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 16:55:37.595966 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:37.595933    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 16:55:38.593211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:38.592889    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 16:55:38.593211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:38.592966    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 16:55:38.899286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:38.899196    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"]
Oct 30 16:55:39.598212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:39.597608    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 16:55:39.598212 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:39.597684    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 16:55:40.597153 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:40.597114    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 16:55:40.597390 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:40.597174    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 16:55:41.597980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:41.597949    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:55:41.599253 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:41.599229    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 16:55:43.900495 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:43.900453    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp" podUID="2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" containerName="payload-processing" containerID="cri-o://66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f" gracePeriod=25
Oct 30 16:55:44.749298 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:44.749262    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp" podUID="2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 16:55:51.217795 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:51.217764    4497 scope.go:122] "RemoveContainer" containerID="98c0fcf8d4294eb8b1750568622e29e07883a055e9e53a0296a3f584ab918a14"
Oct 30 16:55:51.916570 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:55:51.916533    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:55:53.310955 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.310922    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"]
Oct 30 16:55:53.315424 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.315401    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.343813 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.343785    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bff6e\"/\"kube-root-ca.crt\""
Oct 30 16:55:53.344022 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.344000    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bff6e\"/\"auth-test-model-8bff6e-kserve-self-signed-certs\""
Oct 30 16:55:53.344102 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.344056    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bff6e\"/\"openshift-service-ca.crt\""
Oct 30 16:55:53.344278 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.344263    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8bff6e\"/\"default-dockercfg-zcbzm\""
Oct 30 16:55:53.347619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.347597    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"]
Oct 30 16:55:53.389094 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.389048    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tmp-dir\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.389309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.389167    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-home\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.389309 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.389225    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqh4\" (UniqueName: \"kubernetes.io/projected/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-kube-api-access-whqh4\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.389465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.389362    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-dshm\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.389465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.389453    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-model-cache\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.389540 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.389498    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tls-certs\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.490808 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.490773    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tmp-dir\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.490980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.490862    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-home\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.490980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.490880    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqh4\" (UniqueName: \"kubernetes.io/projected/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-kube-api-access-whqh4\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.490980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.490912    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-dshm\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.490980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.490964    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-model-cache\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.490980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.490978    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tls-certs\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.491286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.491263    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tmp-dir\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.491286 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.491276    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-home\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.491412 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.491387    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-model-cache\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.493271 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.493250    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-dshm\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.493444 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.493428    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tls-certs\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.512840 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.512808    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqh4\" (UniqueName: \"kubernetes.io/projected/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-kube-api-access-whqh4\") pod \"auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") " pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.628082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.627916    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:53.938697 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:53.938616    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"]
Oct 30 16:55:53.944338 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:55:53.944309    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0be0cee1_c371_4bbc_a44e_4ac1e99e4893.slice/crio-d91c88325bd821bcc7752b2fc87b1d3ca43db09034962b94256377bc5cdd7cca WatchSource:0}: Error finding container d91c88325bd821bcc7752b2fc87b1d3ca43db09034962b94256377bc5cdd7cca: Status 404 returned error can't find the container with id d91c88325bd821bcc7752b2fc87b1d3ca43db09034962b94256377bc5cdd7cca
Oct 30 16:55:54.222072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:54.221980    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt" event={"ID":"0be0cee1-c371-4bbc-a44e-4ac1e99e4893","Type":"ContainerStarted","Data":"3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203"}
Oct 30 16:55:54.222072 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:54.222036    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt" event={"ID":"0be0cee1-c371-4bbc-a44e-4ac1e99e4893","Type":"ContainerStarted","Data":"d91c88325bd821bcc7752b2fc87b1d3ca43db09034962b94256377bc5cdd7cca"}
Oct 30 16:55:54.262056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:54.262002    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt" podStartSLOduration=1.2619880430000001 podStartE2EDuration="1.261988043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:54.258975428 +0000 UTC m=+1984.112853693" watchObservedRunningTime="2026-10-30 16:55:54.261988043 +0000 UTC m=+1984.115866310"
Oct 30 16:55:54.628622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:54.628586    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:55:54.749830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:55:54.749783    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp" podUID="2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 16:56:03.628083 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:03.628048    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:56:03.640848 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:03.640819    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:56:04.186869 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.186842    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:56:04.245051 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.245022    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-plugins-config-volume\") pod \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\" (UID: \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\") "
Oct 30 16:56:04.245407 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.245387    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-kube-api-access-bdf5s\" (UniqueName: \"kubernetes.io/projected/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-kube-api-access-bdf5s\") pod \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\" (UID: \"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592\") "
Oct 30 16:56:04.245522 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.245495    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-plugins-config-volume" pod "2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" (UID: "2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:56:04.245948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.245930    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:56:04.247552 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.247532    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-kube-api-access-bdf5s" pod "2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" (UID: "2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592"). InnerVolumeSpecName "kube-api-access-bdf5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:56:04.312888 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.312836    4497 generic.go:356] "Generic (PLEG): container finished" podID="2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" containerID="66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f" exitCode=0
Oct 30 16:56:04.313160 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.313135    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp" event={"ID":"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592","Type":"ContainerDied","Data":"66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f"}
Oct 30 16:56:04.313338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.313250    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp" event={"ID":"2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592","Type":"ContainerDied","Data":"e65f3802001da9d500fd75c84d7a19cad9a41cd62a9af74069613fb4732d4624"}
Oct 30 16:56:04.313338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.313278    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"
Oct 30 16:56:04.313338 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.313321    4497 scope.go:122] "RemoveContainer" containerID="66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f"
Oct 30 16:56:04.325311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.325290    4497 scope.go:122] "RemoveContainer" containerID="66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f"
Oct 30 16:56:04.325627 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:56:04.325606    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f\": container with ID starting with 66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f not found: ID does not exist" containerID="66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f"
Oct 30 16:56:04.325715 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.325634    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f"} err="failed to get container status \"66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f\": rpc error: code = NotFound desc = could not find container \"66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f\": container with ID starting with 66b9b08b59fda65999f73fc8ed3dd9fb9b7b40ba9f0c584a5a86ac965738978f not found: ID does not exist"
Oct 30 16:56:04.326826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.326806    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:56:04.347069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.347042    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdf5s\" (UniqueName: \"kubernetes.io/projected/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592-kube-api-access-bdf5s\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:56:04.363046 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.360549    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"]
Oct 30 16:56:04.365508 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:04.365482    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-07cb3c-7b95f7fcbd-82sgp"]
Oct 30 16:56:06.300654 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:06.300618    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" path="/var/lib/kubelet/pods/2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592/volumes"
Oct 30 16:56:21.916630 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:56:21.916590    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:56:22.375695 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.375660    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"]
Oct 30 16:56:22.376440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.376420    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" containerName="payload-processing"
Oct 30 16:56:22.376440 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.376440    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" containerName="payload-processing"
Oct 30 16:56:22.376648 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.376635    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a13e5cb-1b3b-47fe-8e1f-22cb1cf8e592" containerName="payload-processing"
Oct 30 16:56:22.379652 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.379629    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.392686 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.392661    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-3ca339\"/\"kube-root-ca.crt\""
Oct 30 16:56:22.392827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.392774    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-3ca339\"/\"openshift-service-ca.crt\""
Oct 30 16:56:22.402438 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.402410    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-3ca339\"/\"auth-test-model-3ca339-kserve-self-signed-certs\""
Oct 30 16:56:22.402647 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.402482    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-3ca339\"/\"default-dockercfg-crpg2\""
Oct 30 16:56:22.407845 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.407822    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"]
Oct 30 16:56:22.511381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.511345    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-home\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.511587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.511541    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-model-cache\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.511674 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.511656    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rpcb\" (UniqueName: \"kubernetes.io/projected/a036fa80-7f42-4286-9d47-4e3d250dcb65-kube-api-access-4rpcb\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.511724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.511683    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a036fa80-7f42-4286-9d47-4e3d250dcb65-tls-certs\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.511783 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.511742    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-dshm\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.511843 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.511815    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-tmp-dir\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.613747 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.613714    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-home\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.613966 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.613887    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-model-cache\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.614039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.613975    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rpcb\" (UniqueName: \"kubernetes.io/projected/a036fa80-7f42-4286-9d47-4e3d250dcb65-kube-api-access-4rpcb\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.614110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.614004    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a036fa80-7f42-4286-9d47-4e3d250dcb65-tls-certs\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.614209 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.614179    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-dshm\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.614288 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.614229    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-tmp-dir\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.614288 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.614246    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-home\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.614620 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.614590    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-model-cache\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.614756 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.614678    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-tmp-dir\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.619956 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.619797    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a036fa80-7f42-4286-9d47-4e3d250dcb65-tls-certs\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.634272 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.620583    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-dshm\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.667787 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.667749    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rpcb\" (UniqueName: \"kubernetes.io/projected/a036fa80-7f42-4286-9d47-4e3d250dcb65-kube-api-access-4rpcb\") pod \"auth-test-model-3ca339-kserve-64885cd64f-pt5fp\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") " pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:22.695689 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:22.695640    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:23.019357 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:23.019333    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"]
Oct 30 16:56:23.022495 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:56:23.022467    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda036fa80_7f42_4286_9d47_4e3d250dcb65.slice/crio-ff5a347cf8989132e94d021beb880fcb5c8bc408723d3e5b1fb1d918ea590900 WatchSource:0}: Error finding container ff5a347cf8989132e94d021beb880fcb5c8bc408723d3e5b1fb1d918ea590900: Status 404 returned error can't find the container with id ff5a347cf8989132e94d021beb880fcb5c8bc408723d3e5b1fb1d918ea590900
Oct 30 16:56:23.457880 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:23.457784    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp" event={"ID":"a036fa80-7f42-4286-9d47-4e3d250dcb65","Type":"ContainerStarted","Data":"d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d"}
Oct 30 16:56:23.457880 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:23.457839    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp" event={"ID":"a036fa80-7f42-4286-9d47-4e3d250dcb65","Type":"ContainerStarted","Data":"ff5a347cf8989132e94d021beb880fcb5c8bc408723d3e5b1fb1d918ea590900"}
Oct 30 16:56:23.497866 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:23.497791    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp" podStartSLOduration=1.497769995 podStartE2EDuration="1.497769995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:23.494403995 +0000 UTC m=+2013.348282259" watchObservedRunningTime="2026-10-30 16:56:23.497769995 +0000 UTC m=+2013.351648260"
Oct 30 16:56:23.696558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:23.696523    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:32.696532 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:32.696452    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:32.709483 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:32.709457    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:33.545327 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:56:33.545294    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 16:56:51.918929 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:56:51.918882    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:57:21.919377 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:57:21.919338    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:57:45.714243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:45.714207    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"]
Oct 30 16:57:45.714971 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:45.714845    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy" containerID="cri-o://b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7" gracePeriod=30
Oct 30 16:57:46.985282 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:46.985242    4497 patch_prober.go:29] interesting pod/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:57:46.985698 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:46.985325    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:57:50.994871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:50.994806    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:57:51.115179 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115147    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-certs\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.115415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115224    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-data\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.115415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115292    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-kube-api-access-xlbq4\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-kube-api-access-xlbq4\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.115415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115321    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42284a65-b5fb-4b79-b717-1961164e79ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42284a65-b5fb-4b79-b717-1961164e79ce-istio-podinfo\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.115415 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115342    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-credential-socket\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.115665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115500    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-certs" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:57:51.115665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115548    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-data" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:57:51.115665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115553    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42284a65-b5fb-4b79-b717-1961164e79ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42284a65-b5fb-4b79-b717-1961164e79ce-istiod-ca-cert\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.115841 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115700    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-credential-socket" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:57:51.115841 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115704    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-istio-token\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-istio-token\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.115841 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115788    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-socket\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.115994 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115852    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42284a65-b5fb-4b79-b717-1961164e79ce-istiod-ca-cert" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:57:51.115994 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.115866    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-envoy\") pod \"42284a65-b5fb-4b79-b717-1961164e79ce\" (UID: \"42284a65-b5fb-4b79-b717-1961164e79ce\") "
Oct 30 16:57:51.116100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.116057    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-socket" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:57:51.116628 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.116606    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.116739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.116630    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.116739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.116643    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42284a65-b5fb-4b79-b717-1961164e79ce-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.116739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.116656    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.116739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.116667    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.118375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.118338    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42284a65-b5fb-4b79-b717-1961164e79ce-istio-podinfo" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:57:51.118375 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.118355    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-envoy" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:57:51.118584 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.118562    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-kube-api-access-xlbq4" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "kube-api-access-xlbq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:57:51.118690 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.118659    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-istio-token" pod "42284a65-b5fb-4b79-b717-1961164e79ce" (UID: "42284a65-b5fb-4b79-b717-1961164e79ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:57:51.123347 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.123321    4497 generic.go:356] "Generic (PLEG): container finished" podID="42284a65-b5fb-4b79-b717-1961164e79ce" containerID="b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7" exitCode=0
Oct 30 16:57:51.123473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.123395    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"
Oct 30 16:57:51.123473 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.123418    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" event={"ID":"42284a65-b5fb-4b79-b717-1961164e79ce","Type":"ContainerDied","Data":"b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7"}
Oct 30 16:57:51.123643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.123542    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8" event={"ID":"42284a65-b5fb-4b79-b717-1961164e79ce","Type":"ContainerDied","Data":"a8f4a83d7e6b269e15df928b193a229e22f8d773d3883852d7ee89dc06f3b754"}
Oct 30 16:57:51.123643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.123583    4497 scope.go:122] "RemoveContainer" containerID="b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7"
Oct 30 16:57:51.153560 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.153534    4497 scope.go:122] "RemoveContainer" containerID="6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90"
Oct 30 16:57:51.163506 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.163485    4497 scope.go:122] "RemoveContainer" containerID="b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7"
Oct 30 16:57:51.163783 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:57:51.163763    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7\": container with ID starting with b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7 not found: ID does not exist" containerID="b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7"
Oct 30 16:57:51.163834 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.163795    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7"} err="failed to get container status \"b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7\": rpc error: code = NotFound desc = could not find container \"b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7\": container with ID starting with b192eb4757cad27010e09102dee30d25cff95d0189b2125ca4f399856896ebc7 not found: ID does not exist"
Oct 30 16:57:51.163834 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.163815    4497 scope.go:122] "RemoveContainer" containerID="6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90"
Oct 30 16:57:51.164037 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:57:51.164019    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90\": container with ID starting with 6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90 not found: ID does not exist" containerID="6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90"
Oct 30 16:57:51.164081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.164043    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90"} err="failed to get container status \"6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90\": rpc error: code = NotFound desc = could not find container \"6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90\": container with ID starting with 6e73424c23acce3fc561f2ea1b3b5a6c4e30a8761d54e85904a78638ca177d90 not found: ID does not exist"
Oct 30 16:57:51.187632 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.187600    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"]
Oct 30 16:57:51.191064 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.191037    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-07cb3c-openshift-default-66d56db8f-cxtz8"]
Oct 30 16:57:51.217645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.217617    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.217645 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.217642    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42284a65-b5fb-4b79-b717-1961164e79ce-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.217802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.217652    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlbq4\" (UniqueName: \"kubernetes.io/projected/42284a65-b5fb-4b79-b717-1961164e79ce-kube-api-access-xlbq4\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.217802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:51.217661    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42284a65-b5fb-4b79-b717-1961164e79ce-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:57:51.918016 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:57:51.917983    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:57:52.301437 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:57:52.301405    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" path="/var/lib/kubelet/pods/42284a65-b5fb-4b79-b717-1961164e79ce/volumes"
Oct 30 16:58:10.580970 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:10.580780    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"]
Oct 30 16:58:11.262724 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.262654    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh" podUID="651fa5cf-eaa0-4fb8-965e-48bd9564d79b" containerName="maas-api" containerID="cri-o://799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb" gracePeriod=30
Oct 30 16:58:11.558926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.558893    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:58:11.720646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.720602    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-kube-api-access-77rjq\" (UniqueName: \"kubernetes.io/projected/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-kube-api-access-77rjq\") pod \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") "
Oct 30 16:58:11.722674 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.722643    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-openshift-service-ca\") pod \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") "
Oct 30 16:58:11.722942 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.722919    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls\") pod \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") "
Oct 30 16:58:11.723048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.722995    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-metrics-certs\") pod \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\" (UID: \"651fa5cf-eaa0-4fb8-965e-48bd9564d79b\") "
Oct 30 16:58:11.723159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.723130    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-openshift-service-ca" pod "651fa5cf-eaa0-4fb8-965e-48bd9564d79b" (UID: "651fa5cf-eaa0-4fb8-965e-48bd9564d79b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:58:11.723815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.723794    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:11.724335 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.724309    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-kube-api-access-77rjq" pod "651fa5cf-eaa0-4fb8-965e-48bd9564d79b" (UID: "651fa5cf-eaa0-4fb8-965e-48bd9564d79b"). InnerVolumeSpecName "kube-api-access-77rjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:11.726091 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.726070    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls" pod "651fa5cf-eaa0-4fb8-965e-48bd9564d79b" (UID: "651fa5cf-eaa0-4fb8-965e-48bd9564d79b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:58:11.726220 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.726202    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-metrics-certs" pod "651fa5cf-eaa0-4fb8-965e-48bd9564d79b" (UID: "651fa5cf-eaa0-4fb8-965e-48bd9564d79b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:58:11.825292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.825173    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:11.825292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.825244    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77rjq\" (UniqueName: \"kubernetes.io/projected/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-kube-api-access-77rjq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:11.825292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:11.825255    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651fa5cf-eaa0-4fb8-965e-48bd9564d79b-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:12.270752 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.270712    4497 generic.go:356] "Generic (PLEG): container finished" podID="651fa5cf-eaa0-4fb8-965e-48bd9564d79b" containerID="799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb" exitCode=0
Oct 30 16:58:12.270957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.270761    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"
Oct 30 16:58:12.270957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.270797    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh" event={"ID":"651fa5cf-eaa0-4fb8-965e-48bd9564d79b","Type":"ContainerDied","Data":"799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb"}
Oct 30 16:58:12.270957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.270878    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh" event={"ID":"651fa5cf-eaa0-4fb8-965e-48bd9564d79b","Type":"ContainerDied","Data":"c45abe2153d94fabf084e74b1d3fb8d551cf2c3aa2ad8abf5656e1353005ca67"}
Oct 30 16:58:12.270957 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.270902    4497 scope.go:122] "RemoveContainer" containerID="799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb"
Oct 30 16:58:12.281687 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.281669    4497 scope.go:122] "RemoveContainer" containerID="799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb"
Oct 30 16:58:12.281981 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:58:12.281962    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb\": container with ID starting with 799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb not found: ID does not exist" containerID="799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb"
Oct 30 16:58:12.282031 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.281992    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb"} err="failed to get container status \"799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb\": rpc error: code = NotFound desc = could not find container \"799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb\": container with ID starting with 799d9d0ca484c02682282cb2b8d271aaa3f64e9c606ad21ff383340a00114ddb not found: ID does not exist"
Oct 30 16:58:12.317542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.317511    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"]
Oct 30 16:58:12.326817 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:12.326791    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77f4f4-7bb469f789-g45gh"]
Oct 30 16:58:14.300977 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:14.300943    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651fa5cf-eaa0-4fb8-965e-48bd9564d79b" path="/var/lib/kubelet/pods/651fa5cf-eaa0-4fb8-965e-48bd9564d79b/volumes"
Oct 30 16:58:21.917511 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:58:21.917464    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:58:46.876646 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:46.876606    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"]
Oct 30 16:58:46.877281 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:46.876954    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy" containerID="cri-o://de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079" gracePeriod=30
Oct 30 16:58:46.886159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:46.886131    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"]
Oct 30 16:58:46.886414 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:46.886376    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy" containerID="cri-o://7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4" gracePeriod=30
Oct 30 16:58:46.890069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:46.890044    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"]
Oct 30 16:58:46.890289 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:46.890253    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy" containerID="cri-o://0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460" gracePeriod=30
Oct 30 16:58:46.893463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:46.893441    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"]
Oct 30 16:58:46.893624 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:46.893604    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy" containerID="cri-o://ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a" gracePeriod=30
Oct 30 16:58:51.923072 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:58:51.923033    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:58:52.223958 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.223929    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:58:52.248758 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.248730    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-kube-api-access-wsnn2\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-kube-api-access-wsnn2\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.248883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.248803    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-envoy\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.248883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.248847    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a9d517-27d1-4c1b-b29a-1d413495d659-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a9d517-27d1-4c1b-b29a-1d413495d659-istiod-ca-cert\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.248982 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.248909    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-credential-socket\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.249046 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249011    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-certs\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.249105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249048    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-socket\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.249105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249092    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-podinfo\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.249243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249177    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-token\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-token\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.249313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249266    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:58:52.249509 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249318    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-data\") pod \"b7a9d517-27d1-4c1b-b29a-1d413495d659\" (UID: \"b7a9d517-27d1-4c1b-b29a-1d413495d659\") "
Oct 30 16:58:52.249604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249365    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-certs" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.249658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249413    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-credential-socket" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.249658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249471    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a9d517-27d1-4c1b-b29a-1d413495d659-istiod-ca-cert" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:58:52.249658 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249565    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-data" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.249872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.249843    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-socket" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.250452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.250206    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7a9d517-27d1-4c1b-b29a-1d413495d659-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.250452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.250230    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.250452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.250245    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.250452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.250258    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.250452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.250271    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.251213 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.251162    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-envoy" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.251586 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.251568    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-kube-api-access-wsnn2" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "kube-api-access-wsnn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:52.251952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.251927    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-token" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:52.252264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.252234    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-podinfo" pod "b7a9d517-27d1-4c1b-b29a-1d413495d659" (UID: "b7a9d517-27d1-4c1b-b29a-1d413495d659"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:58:52.254662 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.254642    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:58:52.258082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.258065    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:58:52.351887 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.351827    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-token\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-token\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.352069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.351915    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-certs\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.352069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.351945    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-socket\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.352069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.351985    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-kube-api-access-rnsjj\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-kube-api-access-rnsjj\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.352069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352023    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-token\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-token\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.352069 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352054    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-data\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.352402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352088    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-data\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.352402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352120    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-podinfo\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.352402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352158    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-data\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.352402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352222    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-certs\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.352402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352258    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-credential-socket\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.352402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352326    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-credential-socket\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.352402 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352385    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-podinfo\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.352755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352423    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istiod-ca-cert\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.352755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352477    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-envoy\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.352755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352515    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6abaa3-2dfe-4e36-bb38-700052316a62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6abaa3-2dfe-4e36-bb38-700052316a62-istiod-ca-cert\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.352755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352551    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325b5a92-0589-4e86-afba-6daf4b04ebd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b5a92-0589-4e86-afba-6daf4b04ebd4-istiod-ca-cert\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.352755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352578    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-certs\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.352755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352663    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-data" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.352755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352701    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-podinfo\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.352755 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352730    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-credential-socket" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.353643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352918    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-certs" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.353643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352956    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-credential-socket" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.353643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.353120    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-data" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.353643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.352696    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-certs" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.353643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.353357    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-socket" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.353643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.353566    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-data" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.353988 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.353896    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istiod-ca-cert" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:58:52.354334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354308    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6abaa3-2dfe-4e36-bb38-700052316a62-istiod-ca-cert" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:58:52.354514 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354493    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-credential-socket\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.354597 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354553    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-socket\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.354659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354601    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-token\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-token\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.354659 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354650    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-kube-api-access-nbvsn\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-kube-api-access-nbvsn\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.354770 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354685    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-envoy\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.354770 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354715    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-envoy\") pod \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\" (UID: \"19ef69a4-f3c8-46ba-9de1-b151e7837edb\") "
Oct 30 16:58:52.354889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354777    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-kube-api-access-w9zbr\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-kube-api-access-w9zbr\") pod \"325b5a92-0589-4e86-afba-6daf4b04ebd4\" (UID: \"325b5a92-0589-4e86-afba-6daf4b04ebd4\") "
Oct 30 16:58:52.354889 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.354835    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-socket\") pod \"3c6abaa3-2dfe-4e36-bb38-700052316a62\" (UID: \"3c6abaa3-2dfe-4e36-bb38-700052316a62\") "
Oct 30 16:58:52.355602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355064    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-certs" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.355602 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355351    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325b5a92-0589-4e86-afba-6daf4b04ebd4-istiod-ca-cert" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:58:52.355808 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355781    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355812    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355828    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355843    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c6abaa3-2dfe-4e36-bb38-700052316a62-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355857    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/325b5a92-0589-4e86-afba-6daf4b04ebd4-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355871    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355887    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsnn2\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-kube-api-access-wsnn2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355902    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355917    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.355930 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355931    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.356449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355945    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.356449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355959    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.356449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355974    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7a9d517-27d1-4c1b-b29a-1d413495d659-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.356449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355988    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.356449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.356003    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.356449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.356016    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.356449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.355869    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-kube-api-access-rnsjj" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "kube-api-access-rnsjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:52.356449 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.356258    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-socket" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.356848 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.356679    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-socket" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.356901 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.356886    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-credential-socket" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.357913 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.357864    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-podinfo" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:58:52.359519 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.359495    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-podinfo" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:58:52.359871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.359847    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-token" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:52.359948 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.359896    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-token" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:52.360138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.360115    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-envoy" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.360419 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.360383    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-podinfo" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:58:52.360619 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.360599    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-token" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:52.360891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.360861    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-kube-api-access-nbvsn" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "kube-api-access-nbvsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:52.360891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.360865    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-envoy" pod "19ef69a4-f3c8-46ba-9de1-b151e7837edb" (UID: "19ef69a4-f3c8-46ba-9de1-b151e7837edb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.360891 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.360871    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-envoy" pod "3c6abaa3-2dfe-4e36-bb38-700052316a62" (UID: "3c6abaa3-2dfe-4e36-bb38-700052316a62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:58:52.361457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.361428    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-kube-api-access-w9zbr" pod "325b5a92-0589-4e86-afba-6daf4b04ebd4" (UID: "325b5a92-0589-4e86-afba-6daf4b04ebd4"). InnerVolumeSpecName "kube-api-access-w9zbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457447    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9zbr\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-kube-api-access-w9zbr\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457479    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457489    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457499    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsjj\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-kube-api-access-rnsjj\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457507    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457517    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457525    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457533    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457541    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.457542 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457549    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/325b5a92-0589-4e86-afba-6daf4b04ebd4-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.458205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457557    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.458205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457565    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/325b5a92-0589-4e86-afba-6daf4b04ebd4-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.458205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457573    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbvsn\" (UniqueName: \"kubernetes.io/projected/19ef69a4-f3c8-46ba-9de1-b151e7837edb-kube-api-access-nbvsn\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.458205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457581    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c6abaa3-2dfe-4e36-bb38-700052316a62-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.458205 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.457590    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ef69a4-f3c8-46ba-9de1-b151e7837edb-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:58:52.573128 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.573094    4497 generic.go:356] "Generic (PLEG): container finished" podID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerID="de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079" exitCode=0
Oct 30 16:58:52.573334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.573135    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"
Oct 30 16:58:52.573334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.573176    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" event={"ID":"b7a9d517-27d1-4c1b-b29a-1d413495d659","Type":"ContainerDied","Data":"de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079"}
Oct 30 16:58:52.573334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.573243    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x" event={"ID":"b7a9d517-27d1-4c1b-b29a-1d413495d659","Type":"ContainerDied","Data":"9795407efb6947e62eab65228b6564d7dd687606b2e648fd789cc9f92d0311b1"}
Oct 30 16:58:52.573334 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.573270    4497 scope.go:122] "RemoveContainer" containerID="de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079"
Oct 30 16:58:52.574772 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.574750    4497 generic.go:356] "Generic (PLEG): container finished" podID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerID="7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4" exitCode=0
Oct 30 16:58:52.574871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.574800    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"
Oct 30 16:58:52.574871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.574844    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" event={"ID":"3c6abaa3-2dfe-4e36-bb38-700052316a62","Type":"ContainerDied","Data":"7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4"}
Oct 30 16:58:52.574984 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.574881    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm" event={"ID":"3c6abaa3-2dfe-4e36-bb38-700052316a62","Type":"ContainerDied","Data":"da85e3004312d79ad53c031e1addb44305f27485c4525f358343caa7531e3d29"}
Oct 30 16:58:52.576575 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.576554    4497 generic.go:356] "Generic (PLEG): container finished" podID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerID="ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a" exitCode=0
Oct 30 16:58:52.576709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.576592    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"
Oct 30 16:58:52.576709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.576633    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" event={"ID":"19ef69a4-f3c8-46ba-9de1-b151e7837edb","Type":"ContainerDied","Data":"ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a"}
Oct 30 16:58:52.576709 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.576692    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8" event={"ID":"19ef69a4-f3c8-46ba-9de1-b151e7837edb","Type":"ContainerDied","Data":"3df195b4ed7b9d999072b5d328c2d8f285b2418b981cee7c9c7779bed7da384a"}
Oct 30 16:58:52.578163 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.578133    4497 generic.go:356] "Generic (PLEG): container finished" podID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerID="0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460" exitCode=0
Oct 30 16:58:52.578280 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.578161    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" event={"ID":"325b5a92-0589-4e86-afba-6daf4b04ebd4","Type":"ContainerDied","Data":"0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460"}
Oct 30 16:58:52.578280 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.578203    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"
Oct 30 16:58:52.578280 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.578222    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj" event={"ID":"325b5a92-0589-4e86-afba-6daf4b04ebd4","Type":"ContainerDied","Data":"e94520e846129cf8928a7ded41ad4a206d2ef260a3e162ba12c5877a4e56c903"}
Oct 30 16:58:52.586084 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.586065    4497 scope.go:122] "RemoveContainer" containerID="de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079"
Oct 30 16:58:52.586530 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:58:52.586502    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079\": container with ID starting with de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079 not found: ID does not exist" containerID="de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079"
Oct 30 16:58:52.586603 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.586537    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079"} err="failed to get container status \"de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079\": rpc error: code = NotFound desc = could not find container \"de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079\": container with ID starting with de638a5e71667b95d2c44a6de491a18ee5180b2afb4e7eb3ea444b8fe281a079 not found: ID does not exist"
Oct 30 16:58:52.586603 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.586561    4497 scope.go:122] "RemoveContainer" containerID="7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4"
Oct 30 16:58:52.596871 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.596855    4497 scope.go:122] "RemoveContainer" containerID="7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4"
Oct 30 16:58:52.597140 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:58:52.597118    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4\": container with ID starting with 7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4 not found: ID does not exist" containerID="7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4"
Oct 30 16:58:52.597211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.597147    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4"} err="failed to get container status \"7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4\": rpc error: code = NotFound desc = could not find container \"7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4\": container with ID starting with 7c737684be8abee2e601e9d5bee8ff5a904cfd5042228f7a57e3fd50d9e91fb4 not found: ID does not exist"
Oct 30 16:58:52.597211 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.597163    4497 scope.go:122] "RemoveContainer" containerID="ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a"
Oct 30 16:58:52.606024 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.606002    4497 scope.go:122] "RemoveContainer" containerID="ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a"
Oct 30 16:58:52.606314 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:58:52.606293    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a\": container with ID starting with ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a not found: ID does not exist" containerID="ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a"
Oct 30 16:58:52.606450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.606320    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a"} err="failed to get container status \"ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a\": rpc error: code = NotFound desc = could not find container \"ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a\": container with ID starting with ca4ee7f0d9ad09b53dd54d8a4a9e4cadc6a83135aa8da8354b2230984d17066a not found: ID does not exist"
Oct 30 16:58:52.606450 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.606335    4497 scope.go:122] "RemoveContainer" containerID="0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460"
Oct 30 16:58:52.615129 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.615110    4497 scope.go:122] "RemoveContainer" containerID="0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460"
Oct 30 16:58:52.615481 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:58:52.615462    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460\": container with ID starting with 0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460 not found: ID does not exist" containerID="0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460"
Oct 30 16:58:52.615538 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.615491    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460"} err="failed to get container status \"0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460\": rpc error: code = NotFound desc = could not find container \"0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460\": container with ID starting with 0fb4a31fafd7d368757e5fe9de3295b4868f6bbc84eae92fb4f5b9f3248c0460 not found: ID does not exist"
Oct 30 16:58:52.631663 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.631638    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"]
Oct 30 16:58:52.637216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.637158    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2cb2x"]
Oct 30 16:58:52.693081 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.693044    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"]
Oct 30 16:58:52.696743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.696719    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-qxhdj"]
Oct 30 16:58:52.726397 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.726310    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"]
Oct 30 16:58:52.730716 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.730419    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-dt5q8"]
Oct 30 16:58:52.759498 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.759463    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"]
Oct 30 16:58:52.763541 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:52.763516    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-khntm"]
Oct 30 16:58:53.209025 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:53.208995    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"]
Oct 30 16:58:54.276243 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.276211    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"]
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.276890    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.276911    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.276927    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.276935    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.276977    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.276985    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277006    4497 cpu_manager.go:414] "containerMap: removing container" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277013    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277021    4497 cpu_manager.go:414] "containerMap: removing container" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277027    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277048    4497 cpu_manager.go:414] "containerMap: removing container" podUID="651fa5cf-eaa0-4fb8-965e-48bd9564d79b" containerName="maas-api"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277054    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651fa5cf-eaa0-4fb8-965e-48bd9564d79b" containerName="maas-api"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277240    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277256    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277266    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277279    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277287    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" containerName="istio-proxy"
Oct 30 16:58:54.277448 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.277297    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="651fa5cf-eaa0-4fb8-965e-48bd9564d79b" containerName="maas-api"
Oct 30 16:58:54.282721 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.282700    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.295860 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.295832    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-2035f6-openshift-default-dockercfg-8xzb7\""
Oct 30 16:58:54.307233 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.307180    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ef69a4-f3c8-46ba-9de1-b151e7837edb" path="/var/lib/kubelet/pods/19ef69a4-f3c8-46ba-9de1-b151e7837edb/volumes"
Oct 30 16:58:54.307921 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.307897    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325b5a92-0589-4e86-afba-6daf4b04ebd4" path="/var/lib/kubelet/pods/325b5a92-0589-4e86-afba-6daf4b04ebd4/volumes"
Oct 30 16:58:54.308607 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.308576    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6abaa3-2dfe-4e36-bb38-700052316a62" path="/var/lib/kubelet/pods/3c6abaa3-2dfe-4e36-bb38-700052316a62/volumes"
Oct 30 16:58:54.309718 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.309665    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a9d517-27d1-4c1b-b29a-1d413495d659" path="/var/lib/kubelet/pods/b7a9d517-27d1-4c1b-b29a-1d413495d659/volumes"
Oct 30 16:58:54.310110 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.310089    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"]
Oct 30 16:58:54.385066 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.385026    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-podinfo\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.385241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.385220    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-data\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.385504 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.385481    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-token\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.385581 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.385513    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istiod-ca-cert\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.385627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.385575    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-socket\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.385736 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.385713    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-envoy\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.385883 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.385859    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs876\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-kube-api-access-zs876\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.385923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.385895    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-certs\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.386062 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.386044    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-credential-socket\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.487686    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-token\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.487738    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istiod-ca-cert\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.487772    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-socket\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.487825    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-envoy\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.487912    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs876\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-kube-api-access-zs876\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.487938    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-certs\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.488057    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-credential-socket\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.488094    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-podinfo\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488222 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.488224    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-data\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.488766 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.488732    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-data\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.489545 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.489517    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-certs\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.489725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.489585    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-socket\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.489835 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.489748    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istiod-ca-cert\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.490333 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.490306    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-credential-socket\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.496842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.495965    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-envoy\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.496842 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.496573    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-podinfo\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.510060 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.510034    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-token\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.510627 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.510601    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs876\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-kube-api-access-zs876\") pod \"e2e-noten-2035f6-openshift-default-74cff4f697-7prpx\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") " pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.600251 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.600159    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:54.697292 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.697258    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"]
Oct 30 16:58:54.698276 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.698257    4497 cpu_manager.go:414] "containerMap: removing container" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy"
Oct 30 16:58:54.698337 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.698280    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy"
Oct 30 16:58:54.698467 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.698453    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="42284a65-b5fb-4b79-b717-1961164e79ce" containerName="istio-proxy"
Oct 30 16:58:54.703637 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.703609    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.729571 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.729541    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-18fb02-serving-cert\""
Oct 30 16:58:54.743233 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.742980    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"]
Oct 30 16:58:54.792601 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.792555    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-metrics-certs\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.792772 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.792710    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92cqz\" (UniqueName: \"kubernetes.io/projected/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-kube-api-access-92cqz\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.792827 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.792765    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-openshift-service-ca\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.792952 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.792924    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-maas-api-tls\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.894318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.894224    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-metrics-certs\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.894318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.894285    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92cqz\" (UniqueName: \"kubernetes.io/projected/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-kube-api-access-92cqz\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.894318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.894317    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-openshift-service-ca\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.894605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.894376    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-maas-api-tls\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.895045 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.895019    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-openshift-service-ca\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.896807 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.896784    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-metrics-certs\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.896916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.896836    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-maas-api-tls\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.915928 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.915896    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92cqz\" (UniqueName: \"kubernetes.io/projected/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-kube-api-access-92cqz\") pod \"maas-api-e2e-explicit-18fb02-5698758f64-vmb2m\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:54.932431 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.932408    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"]
Oct 30 16:58:54.934853 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:58:54.934827    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6540cea8_117c_4df9_b2b9_42dc45f8fbc4.slice/crio-fd358db0bc3c83c3819cc38823e0665a8f00f17098fb5ea82aaedd1ba6a3d669 WatchSource:0}: Error finding container fd358db0bc3c83c3819cc38823e0665a8f00f17098fb5ea82aaedd1ba6a3d669: Status 404 returned error can't find the container with id fd358db0bc3c83c3819cc38823e0665a8f00f17098fb5ea82aaedd1ba6a3d669
Oct 30 16:58:54.937523 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.937492    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:58:54.937604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.937557    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:58:54.937604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:54.937586    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:58:55.035430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:55.035382    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:55.358019 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:55.357960    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"]
Oct 30 16:58:55.360839 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:58:55.360809    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb39e43_ec8d_4b5f_a850_0e49c32c3a07.slice/crio-27b10c730ed0f624327cfdfdba74220b12fc0a14c0a1f352d22c15e1551764ba WatchSource:0}: Error finding container 27b10c730ed0f624327cfdfdba74220b12fc0a14c0a1f352d22c15e1551764ba: Status 404 returned error can't find the container with id 27b10c730ed0f624327cfdfdba74220b12fc0a14c0a1f352d22c15e1551764ba
Oct 30 16:58:55.362862 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:55.362841    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 16:58:55.603397 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:55.603364    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx" event={"ID":"6540cea8-117c-4df9-b2b9-42dc45f8fbc4","Type":"ContainerStarted","Data":"58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783"}
Oct 30 16:58:55.603588 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:55.603418    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx" event={"ID":"6540cea8-117c-4df9-b2b9-42dc45f8fbc4","Type":"ContainerStarted","Data":"fd358db0bc3c83c3819cc38823e0665a8f00f17098fb5ea82aaedd1ba6a3d669"}
Oct 30 16:58:55.604694 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:55.604672    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m" event={"ID":"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07","Type":"ContainerStarted","Data":"27b10c730ed0f624327cfdfdba74220b12fc0a14c0a1f352d22c15e1551764ba"}
Oct 30 16:58:55.665416 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:55.665310    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx" podStartSLOduration=1.66529467 podStartE2EDuration="1.66529467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:55.653385869 +0000 UTC m=+2165.507264133" watchObservedRunningTime="2026-10-30 16:58:55.66529467 +0000 UTC m=+2165.519172934"
Oct 30 16:58:56.600781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:56.600745    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:56.606011 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:56.605989    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:56.612969 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:56.612936    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m" event={"ID":"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07","Type":"ContainerStarted","Data":"037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d"}
Oct 30 16:58:56.613127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:56.613076    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:58:56.613127 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:56.613107    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:56.614016 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:56.613997    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:58:56.689048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:58:56.688993    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m" podStartSLOduration=2.011463424 podStartE2EDuration="2.688975818s" totalImagesPullingTime="677.512394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:56.687324058 +0000 UTC m=+2166.541202323" watchObservedRunningTime="2026-10-30 16:58:56.688975818 +0000 UTC m=+2166.542854083"
Oct 30 16:59:02.629289 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:02.629259    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 16:59:08.230422 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.230359    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl" podUID="4b94b372-99c7-4b87-9ecf-f54e87ee055c" containerName="main" containerID="cri-o://fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e" gracePeriod=45
Oct 30 16:59:08.500499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.500435    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:59:08.593326 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.593294    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-home\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-home\") pod \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") "
Oct 30 16:59:08.593503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.593423    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b94b372-99c7-4b87-9ecf-f54e87ee055c-kube-api-access-6k747\" (UniqueName: \"kubernetes.io/projected/4b94b372-99c7-4b87-9ecf-f54e87ee055c-kube-api-access-6k747\") pod \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") "
Oct 30 16:59:08.593503 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.593478    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-model-cache\") pod \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") "
Oct 30 16:59:08.593595 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.593543    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tls-certs\") pod \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") "
Oct 30 16:59:08.593595 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.593575    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tmp-dir\") pod \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") "
Oct 30 16:59:08.593665 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.593617    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-home" pod "4b94b372-99c7-4b87-9ecf-f54e87ee055c" (UID: "4b94b372-99c7-4b87-9ecf-f54e87ee055c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:08.593739 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.593724    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-dshm\") pod \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\" (UID: \"4b94b372-99c7-4b87-9ecf-f54e87ee055c\") "
Oct 30 16:59:08.593855 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.593832    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-model-cache" pod "4b94b372-99c7-4b87-9ecf-f54e87ee055c" (UID: "4b94b372-99c7-4b87-9ecf-f54e87ee055c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:08.594418 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.594300    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:08.594418 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.594322    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:08.594746 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.594722    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tmp-dir" pod "4b94b372-99c7-4b87-9ecf-f54e87ee055c" (UID: "4b94b372-99c7-4b87-9ecf-f54e87ee055c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:08.596740 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.596715    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-dshm" pod "4b94b372-99c7-4b87-9ecf-f54e87ee055c" (UID: "4b94b372-99c7-4b87-9ecf-f54e87ee055c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:08.597008 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.596974    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tls-certs" pod "4b94b372-99c7-4b87-9ecf-f54e87ee055c" (UID: "4b94b372-99c7-4b87-9ecf-f54e87ee055c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:59:08.597688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.597667    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b94b372-99c7-4b87-9ecf-f54e87ee055c-kube-api-access-6k747" pod "4b94b372-99c7-4b87-9ecf-f54e87ee055c" (UID: "4b94b372-99c7-4b87-9ecf-f54e87ee055c"). InnerVolumeSpecName "kube-api-access-6k747". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:08.696029 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.695977    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:08.696235 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.696032    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k747\" (UniqueName: \"kubernetes.io/projected/4b94b372-99c7-4b87-9ecf-f54e87ee055c-kube-api-access-6k747\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:08.696235 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.696055    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:08.696235 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.696069    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b94b372-99c7-4b87-9ecf-f54e87ee055c-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:08.702596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.702564    4497 generic.go:356] "Generic (PLEG): container finished" podID="4b94b372-99c7-4b87-9ecf-f54e87ee055c" containerID="fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e" exitCode=0
Oct 30 16:59:08.702743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.702614    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"
Oct 30 16:59:08.702743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.702658    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl" event={"ID":"4b94b372-99c7-4b87-9ecf-f54e87ee055c","Type":"ContainerDied","Data":"fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e"}
Oct 30 16:59:08.702743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.702707    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl" event={"ID":"4b94b372-99c7-4b87-9ecf-f54e87ee055c","Type":"ContainerDied","Data":"9db57768bc4b097924b6961c26c00cf46b1a5604f7dce9c638ab10bf1a53ffd4"}
Oct 30 16:59:08.702743 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.702724    4497 scope.go:122] "RemoveContainer" containerID="fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e"
Oct 30 16:59:08.713159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.713141    4497 scope.go:122] "RemoveContainer" containerID="fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e"
Oct 30 16:59:08.713454 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:08.713432    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e\": container with ID starting with fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e not found: ID does not exist" containerID="fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e"
Oct 30 16:59:08.713518 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.713460    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e"} err="failed to get container status \"fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e\": rpc error: code = NotFound desc = could not find container \"fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e\": container with ID starting with fd2ae43764572d2cbfd73ccded8d4d56f10e71d6fdf87b83c16648475b81c04e not found: ID does not exist"
Oct 30 16:59:08.748122 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.748090    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"]
Oct 30 16:59:08.752170 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:08.752106    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-deb721/auth-test-model-deb721-kserve-5d5f8b84b9-r7rcl"]
Oct 30 16:59:10.300381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:10.300348    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b94b372-99c7-4b87-9ecf-f54e87ee055c" path="/var/lib/kubelet/pods/4b94b372-99c7-4b87-9ecf-f54e87ee055c/volumes"
Oct 30 16:59:17.480288 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:17.480243    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"]
Oct 30 16:59:17.480781 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:17.480551    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy" containerID="cri-o://d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235" gracePeriod=30
Oct 30 16:59:21.922876 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:21.922832    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:59:22.755778 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.755755    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:59:22.798821 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.798782    4497 generic.go:356] "Generic (PLEG): container finished" podID="22350f1f-626e-41f4-be41-aa75b103daea" containerID="d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235" exitCode=0
Oct 30 16:59:22.798998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.798833    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"
Oct 30 16:59:22.798998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.798854    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" event={"ID":"22350f1f-626e-41f4-be41-aa75b103daea","Type":"ContainerDied","Data":"d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235"}
Oct 30 16:59:22.798998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.798926    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn" event={"ID":"22350f1f-626e-41f4-be41-aa75b103daea","Type":"ContainerDied","Data":"52e9a202ef48d259923023396365b83f50df06cd631e55373430f6574c9b5cc1"}
Oct 30 16:59:22.798998 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.798957    4497 scope.go:122] "RemoveContainer" containerID="d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235"
Oct 30 16:59:22.809157 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.809125    4497 scope.go:122] "RemoveContainer" containerID="d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235"
Oct 30 16:59:22.809494 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:22.809473    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235\": container with ID starting with d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235 not found: ID does not exist" containerID="d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235"
Oct 30 16:59:22.809577 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.809506    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235"} err="failed to get container status \"d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235\": rpc error: code = NotFound desc = could not find container \"d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235\": container with ID starting with d683973c79b69a17e56ad51f994dd6fc719c6788d71cf0b5469dcdfcac88a235 not found: ID does not exist"
Oct 30 16:59:22.827010 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.826972    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-socket\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.827250 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827138    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-credential-socket\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.827250 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827205    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-istio-token\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-istio-token\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.827250 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827235    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-envoy\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.827458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827275    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-socket" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:22.827458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827408    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22350f1f-626e-41f4-be41-aa75b103daea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22350f1f-626e-41f4-be41-aa75b103daea-istiod-ca-cert\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.827458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827411    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-credential-socket" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:22.827458 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827446    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22350f1f-626e-41f4-be41-aa75b103daea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22350f1f-626e-41f4-be41-aa75b103daea-istio-podinfo\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.827677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827486    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-kube-api-access-mpwz4\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-kube-api-access-mpwz4\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.827677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827528    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-data\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.827677 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.827563    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-certs\") pod \"22350f1f-626e-41f4-be41-aa75b103daea\" (UID: \"22350f1f-626e-41f4-be41-aa75b103daea\") "
Oct 30 16:59:22.828552 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.828236    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:22.828552 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.828261    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:22.828552 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.828353    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22350f1f-626e-41f4-be41-aa75b103daea-istiod-ca-cert" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:59:22.828552 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.828484    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-certs" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:22.828850 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.828571    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-data" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:22.831228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.830116    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-istio-token" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:22.831228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.830341    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-envoy" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:22.831228 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.830441    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22350f1f-626e-41f4-be41-aa75b103daea-istio-podinfo" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:59:22.831454 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.831376    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-kube-api-access-mpwz4" pod "22350f1f-626e-41f4-be41-aa75b103daea" (UID: "22350f1f-626e-41f4-be41-aa75b103daea"). InnerVolumeSpecName "kube-api-access-mpwz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:22.930056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.930015    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:22.930056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.930046    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:22.930056 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.930060    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22350f1f-626e-41f4-be41-aa75b103daea-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:22.930604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.930071    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22350f1f-626e-41f4-be41-aa75b103daea-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:22.930604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.930083    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpwz4\" (UniqueName: \"kubernetes.io/projected/22350f1f-626e-41f4-be41-aa75b103daea-kube-api-access-mpwz4\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:22.930604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.930094    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:22.930604 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:22.930107    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22350f1f-626e-41f4-be41-aa75b103daea-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:23.150553 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:23.150519    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"]
Oct 30 16:59:23.159451 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:23.159423    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-wmwzn"]
Oct 30 16:59:24.301053 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:24.301019    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22350f1f-626e-41f4-be41-aa75b103daea" path="/var/lib/kubelet/pods/22350f1f-626e-41f4-be41-aa75b103daea/volumes"
Oct 30 16:59:36.707578 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:36.707484    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"]
Oct 30 16:59:36.708148 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:36.707725    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy" containerID="cri-o://263c87bc90091a33dde6d7caf574bc77782d39fa832a39f8fb474ce191e856af" gracePeriod=30
Oct 30 16:59:40.204961 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:40.204927    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"]
Oct 30 16:59:41.965307 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:41.965277    4497 generic.go:356] "Generic (PLEG): container finished" podID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerID="263c87bc90091a33dde6d7caf574bc77782d39fa832a39f8fb474ce191e856af" exitCode=0
Oct 30 16:59:41.965734 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:41.965353    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" event={"ID":"c5d45ae6-16d3-4605-95ab-19e3b35c4c96","Type":"ContainerDied","Data":"263c87bc90091a33dde6d7caf574bc77782d39fa832a39f8fb474ce191e856af"}
Oct 30 16:59:41.965734 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:41.965442    4497 scope.go:122] "RemoveContainer" containerID="f76a693c91b04fd7917905de51ee79281c802a6013a8b8080b957da62d2445a6"
Oct 30 16:59:41.987722 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:41.987704    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.053897    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istiod-ca-cert\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.056902    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istiod-ca-cert" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.056919    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-socket\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057022    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-certs\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057068    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-podinfo\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057146    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-data\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057241    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-kube-api-access-cjnj9\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-kube-api-access-cjnj9\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057324    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-credential-socket\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057328    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-socket" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057356    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-token\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-token\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057407    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-envoy\") pod \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\" (UID: \"c5d45ae6-16d3-4605-95ab-19e3b35c4c96\") "
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057711    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-credential-socket" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.057951    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-certs" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:42.058429 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.058055    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-data" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:42.061919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.061889    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.062100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.061925    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.062241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.062116    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.062241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.062148    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.062241 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.062162    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.066258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.064394    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-podinfo" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:59:42.066258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.064428    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-kube-api-access-cjnj9" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "kube-api-access-cjnj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:42.066258 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.064625    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-envoy" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:42.066522 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.066355    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-token" pod "c5d45ae6-16d3-4605-95ab-19e3b35c4c96" (UID: "c5d45ae6-16d3-4605-95ab-19e3b35c4c96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:42.163313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.163283    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.163313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.163309    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjnj9\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-kube-api-access-cjnj9\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.163313 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.163321    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.163630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.163329    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d45ae6-16d3-4605-95ab-19e3b35c4c96-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:42.284854 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.284822    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"]
Oct 30 16:59:42.285284 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.285083    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx" podUID="6540cea8-117c-4df9-b2b9-42dc45f8fbc4" containerName="istio-proxy" containerID="cri-o://58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783" gracePeriod=30
Oct 30 16:59:42.973730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.973702    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"
Oct 30 16:59:42.974176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.973703    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh" event={"ID":"c5d45ae6-16d3-4605-95ab-19e3b35c4c96","Type":"ContainerDied","Data":"573678f4e8dcb333c315e426a8c0e8927dba3c28b10904f77450b395a447b6dd"}
Oct 30 16:59:42.974176 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:42.973823    4497 scope.go:122] "RemoveContainer" containerID="263c87bc90091a33dde6d7caf574bc77782d39fa832a39f8fb474ce191e856af"
Oct 30 16:59:43.057232 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:43.057195    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"]
Oct 30 16:59:43.063164 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:43.063139    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-wbxwh"]
Oct 30 16:59:44.300723 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:44.300693    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" path="/var/lib/kubelet/pods/c5d45ae6-16d3-4605-95ab-19e3b35c4c96/volumes"
Oct 30 16:59:47.568216 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.568131    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.641451    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istiod-ca-cert\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.641502    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-envoy\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.641614    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-token\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-token\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.641652    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-kube-api-access-zs876\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-kube-api-access-zs876\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.641715    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-socket\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.641751    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-data\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.641781    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-certs\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.641959    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-credential-socket\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.643006 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.642026    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-podinfo\") pod \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\" (UID: \"6540cea8-117c-4df9-b2b9-42dc45f8fbc4\") "
Oct 30 16:59:47.652287 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.651301    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-podinfo" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:59:47.652881 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.652687    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-socket" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:47.652881 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.652842    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-kube-api-access-zs876" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "kube-api-access-zs876". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:47.653063 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.652926    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-credential-socket" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:47.653316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.653237    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-data" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:47.653316 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.653294    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-certs" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:47.654806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.654781    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istiod-ca-cert" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:59:47.657156 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.657130    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-envoy" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:47.658264 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.658239    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-token" pod "6540cea8-117c-4df9-b2b9-42dc45f8fbc4" (UID: "6540cea8-117c-4df9-b2b9-42dc45f8fbc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:47.744587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744553    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:47.744587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744581    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:47.744587 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744590    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:47.744818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744600    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:47.744818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744610    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:47.744818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744622    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:47.744818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744634    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:47.744818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744642    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:47.744818 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:47.744650    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs876\" (UniqueName: \"kubernetes.io/projected/6540cea8-117c-4df9-b2b9-42dc45f8fbc4-kube-api-access-zs876\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:48.008622 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.008585    4497 generic.go:356] "Generic (PLEG): container finished" podID="6540cea8-117c-4df9-b2b9-42dc45f8fbc4" containerID="58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783" exitCode=0
Oct 30 16:59:48.008802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.008636    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"
Oct 30 16:59:48.008802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.008673    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx" event={"ID":"6540cea8-117c-4df9-b2b9-42dc45f8fbc4","Type":"ContainerDied","Data":"58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783"}
Oct 30 16:59:48.008802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.008747    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx" event={"ID":"6540cea8-117c-4df9-b2b9-42dc45f8fbc4","Type":"ContainerDied","Data":"fd358db0bc3c83c3819cc38823e0665a8f00f17098fb5ea82aaedd1ba6a3d669"}
Oct 30 16:59:48.008802 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.008780    4497 scope.go:122] "RemoveContainer" containerID="58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783"
Oct 30 16:59:48.022308 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.022285    4497 scope.go:122] "RemoveContainer" containerID="58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783"
Oct 30 16:59:48.022658 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:48.022635    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783\": container with ID starting with 58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783 not found: ID does not exist" containerID="58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783"
Oct 30 16:59:48.022737 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.022667    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783"} err="failed to get container status \"58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783\": rpc error: code = NotFound desc = could not find container \"58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783\": container with ID starting with 58447d8ba866653ec183c807662b3569a46c0ca79fa7ed12f8f8a9284fced783 not found: ID does not exist"
Oct 30 16:59:48.054468 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.054437    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"]
Oct 30 16:59:48.064048 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.064019    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-2035f6-openshift-default-74cff4f697-7prpx"]
Oct 30 16:59:48.135830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.135795    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-3b8606"]
Oct 30 16:59:48.136443 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136425    4497 cpu_manager.go:414] "containerMap: removing container" podUID="6540cea8-117c-4df9-b2b9-42dc45f8fbc4" containerName="istio-proxy"
Oct 30 16:59:48.136499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136445    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6540cea8-117c-4df9-b2b9-42dc45f8fbc4" containerName="istio-proxy"
Oct 30 16:59:48.136499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136467    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy"
Oct 30 16:59:48.136499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136472    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy"
Oct 30 16:59:48.136499 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136496    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy"
Oct 30 16:59:48.136630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136501    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy"
Oct 30 16:59:48.136630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136513    4497 cpu_manager.go:414] "containerMap: removing container" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy"
Oct 30 16:59:48.136630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136517    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy"
Oct 30 16:59:48.136630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136525    4497 cpu_manager.go:414] "containerMap: removing container" podUID="4b94b372-99c7-4b87-9ecf-f54e87ee055c" containerName="main"
Oct 30 16:59:48.136630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136530    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b94b372-99c7-4b87-9ecf-f54e87ee055c" containerName="main"
Oct 30 16:59:48.136630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136614    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy"
Oct 30 16:59:48.136630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136624    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="22350f1f-626e-41f4-be41-aa75b103daea" containerName="istio-proxy"
Oct 30 16:59:48.136630 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136632    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b94b372-99c7-4b87-9ecf-f54e87ee055c" containerName="main"
Oct 30 16:59:48.136862 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.136641    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="6540cea8-117c-4df9-b2b9-42dc45f8fbc4" containerName="istio-proxy"
Oct 30 16:59:48.141248 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.141232    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:48.155197 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.155159    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qn5rq\""
Oct 30 16:59:48.158099 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.158075    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-3b8606"]
Oct 30 16:59:48.247596 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.247565    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"]
Oct 30 16:59:48.247777 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.247758    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy" containerID="cri-o://789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638" gracePeriod=30
Oct 30 16:59:48.249916 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.249886    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ksxm\" (UniqueName: \"kubernetes.io/projected/e604b0a8-8210-48a7-80c3-651856bb68c6-kube-api-access-7ksxm\") pod \"test-curl-7437-3b8606\" (UID: \"e604b0a8-8210-48a7-80c3-651856bb68c6\") " pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:48.300688 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.300605    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6540cea8-117c-4df9-b2b9-42dc45f8fbc4" path="/var/lib/kubelet/pods/6540cea8-117c-4df9-b2b9-42dc45f8fbc4/volumes"
Oct 30 16:59:48.351196 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.351159    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ksxm\" (UniqueName: \"kubernetes.io/projected/e604b0a8-8210-48a7-80c3-651856bb68c6-kube-api-access-7ksxm\") pod \"test-curl-7437-3b8606\" (UID: \"e604b0a8-8210-48a7-80c3-651856bb68c6\") " pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:48.372364 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.372333    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ksxm\" (UniqueName: \"kubernetes.io/projected/e604b0a8-8210-48a7-80c3-651856bb68c6-kube-api-access-7ksxm\") pod \"test-curl-7437-3b8606\" (UID: \"e604b0a8-8210-48a7-80c3-651856bb68c6\") " pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:48.453098 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.453064    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:48.760535 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:48.760503    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-3b8606"]
Oct 30 16:59:48.763971 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:59:48.763934    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode604b0a8_8210_48a7_80c3_651856bb68c6.slice/crio-52325ef71e8ef5eddf6e82263ca5e73ff223c2be838976337d43285754d5b17c WatchSource:0}: Error finding container 52325ef71e8ef5eddf6e82263ca5e73ff223c2be838976337d43285754d5b17c: Status 404 returned error can't find the container with id 52325ef71e8ef5eddf6e82263ca5e73ff223c2be838976337d43285754d5b17c
Oct 30 16:59:49.017673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:49.017590    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-3b8606" event={"ID":"e604b0a8-8210-48a7-80c3-651856bb68c6","Type":"ContainerStarted","Data":"694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc"}
Oct 30 16:59:49.017673 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:49.017630    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-3b8606" event={"ID":"e604b0a8-8210-48a7-80c3-651856bb68c6","Type":"ContainerStarted","Data":"52325ef71e8ef5eddf6e82263ca5e73ff223c2be838976337d43285754d5b17c"}
Oct 30 16:59:49.051606 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:49.051551    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7437-3b8606" podStartSLOduration=1.05153777 podStartE2EDuration="1.05153777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:49.050375903 +0000 UTC m=+2218.904254167" watchObservedRunningTime="2026-10-30 16:59:49.05153777 +0000 UTC m=+2218.905416035"
Oct 30 16:59:49.926597 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:49.926559    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-3b8606"]
Oct 30 16:59:49.941699 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:49.941665    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-3b8606"]
Oct 30 16:59:51.030713 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:51.030675    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7437-3b8606" podUID="e604b0a8-8210-48a7-80c3-651856bb68c6" containerName="test-curl-7437-3b8606" containerID="cri-o://694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc" gracePeriod=2
Oct 30 16:59:51.138651 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:51.138617    4497 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 16:59:51.138815 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:51.138669    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 16:59:51.916426 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:51.916384    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 16:59:53.262159 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.262134    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:53.421039 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.421008    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e604b0a8-8210-48a7-80c3-651856bb68c6-kube-api-access-7ksxm\" (UniqueName: \"kubernetes.io/projected/e604b0a8-8210-48a7-80c3-651856bb68c6-kube-api-access-7ksxm\") pod \"e604b0a8-8210-48a7-80c3-651856bb68c6\" (UID: \"e604b0a8-8210-48a7-80c3-651856bb68c6\") "
Oct 30 16:59:53.434291 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.434242    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e604b0a8-8210-48a7-80c3-651856bb68c6-kube-api-access-7ksxm" pod "e604b0a8-8210-48a7-80c3-651856bb68c6" (UID: "e604b0a8-8210-48a7-80c3-651856bb68c6"). InnerVolumeSpecName "kube-api-access-7ksxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:53.497490 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.497464    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:59:53.523465 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.523422    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ksxm\" (UniqueName: \"kubernetes.io/projected/e604b0a8-8210-48a7-80c3-651856bb68c6-kube-api-access-7ksxm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.624605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.624519    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-data\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.624605 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.624586    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-certs\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.624926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.624626    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-token\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-token\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.624926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.624685    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-socket\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.624926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.624767    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-podinfo\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.624926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.624767    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-data" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:53.624926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.624892    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-certs" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:53.624926 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.624908    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-credential-socket\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.625263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625010    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istiod-ca-cert\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.625263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625064    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-socket" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:53.625263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625082    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-kube-api-access-488hf\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-kube-api-access-488hf\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.625263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625113    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-credential-socket" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:53.625263 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625129    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-envoy\") pod \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\" (UID: \"a3508c50-6126-4b95-9b22-ecf6c6d3636c\") "
Oct 30 16:59:53.625536 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625507    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istiod-ca-cert" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 16:59:53.625765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625690    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.625765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625711    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.625765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625726    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.625765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625739    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.625765 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.625753    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.627552 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.627530    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-podinfo" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 16:59:53.627826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.627711    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-envoy" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:53.627826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.627727    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-kube-api-access-488hf" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "kube-api-access-488hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:53.627826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.627749    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-token" pod "a3508c50-6126-4b95-9b22-ecf6c6d3636c" (UID: "a3508c50-6126-4b95-9b22-ecf6c6d3636c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:53.726672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.726640    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.726672 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.726669    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-488hf\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-kube-api-access-488hf\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.726872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.726680    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.726872 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.726690    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3508c50-6126-4b95-9b22-ecf6c6d3636c-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:53.935423 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.935348    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"]
Oct 30 16:59:53.935980 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.935964    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy"
Oct 30 16:59:53.936028 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.935982    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy"
Oct 30 16:59:53.936028 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.936007    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e604b0a8-8210-48a7-80c3-651856bb68c6" containerName="test-curl-7437-3b8606"
Oct 30 16:59:53.936028 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.936012    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e604b0a8-8210-48a7-80c3-651856bb68c6" containerName="test-curl-7437-3b8606"
Oct 30 16:59:53.936124 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.936115    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e604b0a8-8210-48a7-80c3-651856bb68c6" containerName="test-curl-7437-3b8606"
Oct 30 16:59:53.936158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.936132    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d45ae6-16d3-4605-95ab-19e3b35c4c96" containerName="istio-proxy"
Oct 30 16:59:53.936158 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.936143    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerName="istio-proxy"
Oct 30 16:59:53.939903 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.939877    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:53.954806 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.954780    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d5b798-openshift-default-dockercfg-vgn7p\""
Oct 30 16:59:53.968826 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:53.968792    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"]
Oct 30 16:59:54.031036 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.030999    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-certs\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.031036 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.031036    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-credential-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.031311 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.031230    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905c187-904a-44ee-ab84-f6e767fc887b-istio-podinfo\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.031421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.031404    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-istio-token\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.031478 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.031460    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-data\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.031643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.031559    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9kz8\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-kube-api-access-k9kz8\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.031643 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.031589    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-envoy\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.031830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.031680    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.031830 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.031726    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905c187-904a-44ee-ab84-f6e767fc887b-istiod-ca-cert\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.054691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.054297    4497 generic.go:356] "Generic (PLEG): container finished" podID="e604b0a8-8210-48a7-80c3-651856bb68c6" containerID="694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc" exitCode=137
Oct 30 16:59:54.054691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.054432    4497 scope.go:122] "RemoveContainer" containerID="694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc"
Oct 30 16:59:54.054691 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.054562    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.133651    4497 generic.go:356] "Generic (PLEG): container finished" podID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" containerID="789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638" exitCode=0
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.133853    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.133881    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" event={"ID":"a3508c50-6126-4b95-9b22-ecf6c6d3636c","Type":"ContainerDied","Data":"789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638"}
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.133940    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905c187-904a-44ee-ab84-f6e767fc887b-istiod-ca-cert\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.133977    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj" event={"ID":"a3508c50-6126-4b95-9b22-ecf6c6d3636c","Type":"ContainerDied","Data":"69a01a7f69698ff8c0e72adb0ba122416ecea12dd1615ddd5c6d0232a630fc10"}
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.133992    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-certs\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.134031    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-credential-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.134096    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"
Oct 30 16:59:54.134301 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.134225    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905c187-904a-44ee-ab84-f6e767fc887b-istio-podinfo\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.134919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.134359    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-istio-token\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.134919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.134426    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-data\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.134919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.134464    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9kz8\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-kube-api-access-k9kz8\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.134919 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.134507    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-envoy\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.135246 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.135047    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905c187-904a-44ee-ab84-f6e767fc887b-istiod-ca-cert\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.135456 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.135433    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.135901 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.135880    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-data\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.136633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.136610    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-certs\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.136895 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.136857    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-credential-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.138974 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.138943    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-envoy\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.141732 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.141706    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905c187-904a-44ee-ab84-f6e767fc887b-istio-podinfo\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.156321 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.156287    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-istio-token\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.156460 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.156377    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9kz8\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-kube-api-access-k9kz8\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.181138 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.181117    4497 scope.go:122] "RemoveContainer" containerID="694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc"
Oct 30 16:59:54.181570 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:54.181543    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc\": container with ID starting with 694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc not found: ID does not exist" containerID="694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc"
Oct 30 16:59:54.181640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.181580    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc"} err="failed to get container status \"694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc\": rpc error: code = NotFound desc = could not find container \"694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc\": container with ID starting with 694198449fba5c8b882a20098cfea2f2d9053d60742184c189ea26ba86f67ebc not found: ID does not exist"
Oct 30 16:59:54.181640 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.181600    4497 scope.go:122] "RemoveContainer" containerID="789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638"
Oct 30 16:59:54.198215 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.198162    4497 scope.go:122] "RemoveContainer" containerID="789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638"
Oct 30 16:59:54.198647 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:54.198622    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638\": container with ID starting with 789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638 not found: ID does not exist" containerID="789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638"
Oct 30 16:59:54.198796 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.198657    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638"} err="failed to get container status \"789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638\": rpc error: code = NotFound desc = could not find container \"789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638\": container with ID starting with 789f5f35cc5079baa0ba95583f5bafd19f490d6d46def9704d54a7e00a9bf638 not found: ID does not exist"
Oct 30 16:59:54.217924 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.217889    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"]
Oct 30 16:59:54.222168 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.222138    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-w59kj"]
Oct 30 16:59:54.229716 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:54.229681    4497 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7437-3b8606\" is forbidden: User \"system:node:ip-10-0-136-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-74.ec2.internal' and this object" podUID="e604b0a8-8210-48a7-80c3-651856bb68c6" pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:54.240565 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:54.240528    4497 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7437-3b8606\" is forbidden: User \"system:node:ip-10-0-136-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-74.ec2.internal' and this object" podUID="e604b0a8-8210-48a7-80c3-651856bb68c6" pod="opendatahub/test-curl-7437-3b8606"
Oct 30 16:59:54.255029 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.254999    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:54.300788 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.300752    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3508c50-6126-4b95-9b22-ecf6c6d3636c" path="/var/lib/kubelet/pods/a3508c50-6126-4b95-9b22-ecf6c6d3636c/volumes"
Oct 30 16:59:54.301381 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.301360    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e604b0a8-8210-48a7-80c3-651856bb68c6" path="/var/lib/kubelet/pods/e604b0a8-8210-48a7-80c3-651856bb68c6/volumes"
Oct 30 16:59:54.577318 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.577280    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"]
Oct 30 16:59:54.581252 ip-10-0-136-74 kubenswrapper[4497]: W1030 16:59:54.581182    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9905c187_904a_44ee_ab84_f6e767fc887b.slice/crio-07dd5048f59bd59b2467993b876732b8f8ac3b0b2d305e93eaf7c07b57e99415 WatchSource:0}: Error finding container 07dd5048f59bd59b2467993b876732b8f8ac3b0b2d305e93eaf7c07b57e99415: Status 404 returned error can't find the container with id 07dd5048f59bd59b2467993b876732b8f8ac3b0b2d305e93eaf7c07b57e99415
Oct 30 16:59:54.583999 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.583962    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:59:54.584100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.584032    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:59:54.584100 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:54.584063    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 16:59:55.143131 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.143095    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" event={"ID":"9905c187-904a-44ee-ab84-f6e767fc887b","Type":"ContainerStarted","Data":"33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94"}
Oct 30 16:59:55.143380 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.143159    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" event={"ID":"9905c187-904a-44ee-ab84-f6e767fc887b","Type":"ContainerStarted","Data":"07dd5048f59bd59b2467993b876732b8f8ac3b0b2d305e93eaf7c07b57e99415"}
Oct 30 16:59:55.185457 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.185375    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" podStartSLOduration=2.185360001 podStartE2EDuration="2.185360001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:55.183058315 +0000 UTC m=+2225.036936580" watchObservedRunningTime="2026-10-30 16:59:55.185360001 +0000 UTC m=+2225.039238265"
Oct 30 16:59:55.233478 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.233434    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt" podUID="0be0cee1-c371-4bbc-a44e-4ac1e99e4893" containerName="main" containerID="cri-o://3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203" gracePeriod=45
Oct 30 16:59:55.255923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.255898    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:55.261963 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.261940    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:55.314082 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.313940    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt" podUID="0be0cee1-c371-4bbc-a44e-4ac1e99e4893" containerName="main" probeResult="failure" output="Get \"https://10.133.0.159:8000/ready\": dial tcp 10.133.0.159:8000: connect: connection refused"
Oct 30 16:59:55.508962 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.508930    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:59:55.553463 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553429    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tmp-dir\") pod \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") "
Oct 30 16:59:55.553633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553470    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-model-cache\") pod \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") "
Oct 30 16:59:55.553633 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553575    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-kube-api-access-whqh4\" (UniqueName: \"kubernetes.io/projected/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-kube-api-access-whqh4\") pod \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") "
Oct 30 16:59:55.553776 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553693    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-home\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-home\") pod \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") "
Oct 30 16:59:55.553776 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553750    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tmp-dir" pod "0be0cee1-c371-4bbc-a44e-4ac1e99e4893" (UID: "0be0cee1-c371-4bbc-a44e-4ac1e99e4893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:55.553868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553779    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-model-cache" pod "0be0cee1-c371-4bbc-a44e-4ac1e99e4893" (UID: "0be0cee1-c371-4bbc-a44e-4ac1e99e4893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:55.553868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553800    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tls-certs\" (UniqueName: \"kubernetes.io/secret/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tls-certs\") pod \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") "
Oct 30 16:59:55.553868 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553828    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-dshm\") pod \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\" (UID: \"0be0cee1-c371-4bbc-a44e-4ac1e99e4893\") "
Oct 30 16:59:55.554032 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.553944    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-home" pod "0be0cee1-c371-4bbc-a44e-4ac1e99e4893" (UID: "0be0cee1-c371-4bbc-a44e-4ac1e99e4893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:55.554363 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.554341    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:55.554363 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.554360    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:55.554363 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.554369    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:55.556452 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.556419    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tls-certs" pod "0be0cee1-c371-4bbc-a44e-4ac1e99e4893" (UID: "0be0cee1-c371-4bbc-a44e-4ac1e99e4893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 16:59:55.556603 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.556585    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-dshm" pod "0be0cee1-c371-4bbc-a44e-4ac1e99e4893" (UID: "0be0cee1-c371-4bbc-a44e-4ac1e99e4893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 16:59:55.562556 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.562521    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-kube-api-access-whqh4" pod "0be0cee1-c371-4bbc-a44e-4ac1e99e4893" (UID: "0be0cee1-c371-4bbc-a44e-4ac1e99e4893"). InnerVolumeSpecName "kube-api-access-whqh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 16:59:55.655456 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.655421    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whqh4\" (UniqueName: \"kubernetes.io/projected/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-kube-api-access-whqh4\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:55.655456 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.655449    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:55.655456 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:55.655459    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be0cee1-c371-4bbc-a44e-4ac1e99e4893-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 16:59:56.151114 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.151075    4497 generic.go:356] "Generic (PLEG): container finished" podID="0be0cee1-c371-4bbc-a44e-4ac1e99e4893" containerID="3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203" exitCode=0
Oct 30 16:59:56.151358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.151118    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"
Oct 30 16:59:56.151358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.151177    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt" event={"ID":"0be0cee1-c371-4bbc-a44e-4ac1e99e4893","Type":"ContainerDied","Data":"3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203"}
Oct 30 16:59:56.151358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.151249    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt" event={"ID":"0be0cee1-c371-4bbc-a44e-4ac1e99e4893","Type":"ContainerDied","Data":"d91c88325bd821bcc7752b2fc87b1d3ca43db09034962b94256377bc5cdd7cca"}
Oct 30 16:59:56.151358 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.151268    4497 scope.go:122] "RemoveContainer" containerID="3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203"
Oct 30 16:59:56.151580 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.151480    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:56.152730 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.152714    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 16:59:56.161880 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.161858    4497 scope.go:122] "RemoveContainer" containerID="3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203"
Oct 30 16:59:56.162165 ip-10-0-136-74 kubenswrapper[4497]: E1030 16:59:56.162146    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203\": container with ID starting with 3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203 not found: ID does not exist" containerID="3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203"
Oct 30 16:59:56.162234 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.162173    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203"} err="failed to get container status \"3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203\": rpc error: code = NotFound desc = could not find container \"3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203\": container with ID starting with 3476ec5fca26cdfab61c7d7f2e7a3f43231ac420b015299ff48986d208d93203 not found: ID does not exist"
Oct 30 16:59:56.241775 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.241742    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"]
Oct 30 16:59:56.244710 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.244683    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8bff6e/auth-test-model-8bff6e-kserve-75cd8577f7-v7zwt"]
Oct 30 16:59:56.300558 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:56.300525    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be0cee1-c371-4bbc-a44e-4ac1e99e4893" path="/var/lib/kubelet/pods/0be0cee1-c371-4bbc-a44e-4ac1e99e4893/volumes"
Oct 30 16:59:59.572526 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.572485    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"]
Oct 30 16:59:59.573112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.573094    4497 cpu_manager.go:414] "containerMap: removing container" podUID="0be0cee1-c371-4bbc-a44e-4ac1e99e4893" containerName="main"
Oct 30 16:59:59.573112 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.573112    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be0cee1-c371-4bbc-a44e-4ac1e99e4893" containerName="main"
Oct 30 16:59:59.573293 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.573281    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be0cee1-c371-4bbc-a44e-4ac1e99e4893" containerName="main"
Oct 30 16:59:59.577996 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.577977    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 16:59:59.593447 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.593419    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d5b798\""
Oct 30 16:59:59.593599 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.593577    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d5b798-dockercfg-z4vvh\""
Oct 30 16:59:59.596757 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.596734    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"]
Oct 30 16:59:59.759421 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.759371    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77\" (UID: \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 16:59:59.759725 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.759467    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99q2z\" (UniqueName: \"kubernetes.io/projected/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-kube-api-access-99q2z\") pod \"payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77\" (UID: \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 16:59:59.861234 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.861089    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99q2z\" (UniqueName: \"kubernetes.io/projected/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-kube-api-access-99q2z\") pod \"payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77\" (UID: \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 16:59:59.861430 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.861391    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77\" (UID: \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 16:59:59.862105 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.862083    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77\" (UID: \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 16:59:59.882139 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.882108    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99q2z\" (UniqueName: \"kubernetes.io/projected/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-kube-api-access-99q2z\") pod \"payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77\" (UID: \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 16:59:59.891923 ip-10-0-136-74 kubenswrapper[4497]: I1030 16:59:59.891898    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 17:00:00.211434 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.211405    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"]
Oct 30 17:00:00.212347 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:00.212324    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cee62a7_517b_4a8b_8d59_0be8dd2c6d51.slice/crio-fa3016e73ce60650163c291d0c61189da009d2edb7d465478e78a971482d3859 WatchSource:0}: Error finding container fa3016e73ce60650163c291d0c61189da009d2edb7d465478e78a971482d3859: Status 404 returned error can't find the container with id fa3016e73ce60650163c291d0c61189da009d2edb7d465478e78a971482d3859
Oct 30 17:00:00.371841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.371804    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"]
Oct 30 17:00:00.376389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.376343    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:00.397465 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.397419    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"]
Oct 30 17:00:00.402510 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.402481    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"]
Oct 30 17:00:00.402673 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.402596    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:00.409031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.409003    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\""
Oct 30 17:00:00.424437 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.424395    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"]
Oct 30 17:00:00.469989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.469906    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0c469-e42e-40c0-8714-c1771a009900-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr\" (UID: \"e2a0c469-e42e-40c0-8714-c1771a009900\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:00.470125 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.470093    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blhd\" (UniqueName: \"kubernetes.io/projected/e2a0c469-e42e-40c0-8714-c1771a009900-kube-api-access-2blhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr\" (UID: \"e2a0c469-e42e-40c0-8714-c1771a009900\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:00.570916 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.570870    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4k5w\" (UniqueName: \"kubernetes.io/projected/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-kube-api-access-g4k5w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6\" (UID: \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:00.571127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.570926    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0c469-e42e-40c0-8714-c1771a009900-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr\" (UID: \"e2a0c469-e42e-40c0-8714-c1771a009900\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:00.571127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.571031    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6\" (UID: \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:00.571127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.571107    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blhd\" (UniqueName: \"kubernetes.io/projected/e2a0c469-e42e-40c0-8714-c1771a009900-kube-api-access-2blhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr\" (UID: \"e2a0c469-e42e-40c0-8714-c1771a009900\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:00.571635 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.571613    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0c469-e42e-40c0-8714-c1771a009900-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr\" (UID: \"e2a0c469-e42e-40c0-8714-c1771a009900\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:00.592853 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.592824    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blhd\" (UniqueName: \"kubernetes.io/projected/e2a0c469-e42e-40c0-8714-c1771a009900-kube-api-access-2blhd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr\" (UID: \"e2a0c469-e42e-40c0-8714-c1771a009900\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:00.672374 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.672341    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4k5w\" (UniqueName: \"kubernetes.io/projected/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-kube-api-access-g4k5w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6\" (UID: \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:00.672557 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.672390    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6\" (UID: \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:00.673228 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.673204    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6\" (UID: \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:00.693496 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.693468    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4k5w\" (UniqueName: \"kubernetes.io/projected/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-kube-api-access-g4k5w\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6\" (UID: \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:00.700048 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.700013    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:00.722989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:00.722922    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:01.008820 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.008787    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"]
Oct 30 17:00:01.013003 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:01.012964    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a0c469_e42e_40c0_8714_c1771a009900.slice/crio-800aa467405cbaf68ca9a34b58f0961fb1d7ee94966f48fcd3b8cd312dbfb2bb WatchSource:0}: Error finding container 800aa467405cbaf68ca9a34b58f0961fb1d7ee94966f48fcd3b8cd312dbfb2bb: Status 404 returned error can't find the container with id 800aa467405cbaf68ca9a34b58f0961fb1d7ee94966f48fcd3b8cd312dbfb2bb
Oct 30 17:00:01.038577 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.038550    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"]
Oct 30 17:00:01.041138 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:01.041113    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cecf8a_82a6_4a3c_9fd7_e632506c2b45.slice/crio-30e2e34f572bbf9c87cc5ebc2625922dcf2712b755d72466bc92bf52eab2e262 WatchSource:0}: Error finding container 30e2e34f572bbf9c87cc5ebc2625922dcf2712b755d72466bc92bf52eab2e262: Status 404 returned error can't find the container with id 30e2e34f572bbf9c87cc5ebc2625922dcf2712b755d72466bc92bf52eab2e262
Oct 30 17:00:01.185688 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.185645    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr" event={"ID":"e2a0c469-e42e-40c0-8714-c1771a009900","Type":"ContainerStarted","Data":"800aa467405cbaf68ca9a34b58f0961fb1d7ee94966f48fcd3b8cd312dbfb2bb"}
Oct 30 17:00:01.187031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.187004    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6" event={"ID":"97cecf8a-82a6-4a3c-9fd7-e632506c2b45","Type":"ContainerStarted","Data":"30e2e34f572bbf9c87cc5ebc2625922dcf2712b755d72466bc92bf52eab2e262"}
Oct 30 17:00:01.188586 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.188559    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77" event={"ID":"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51","Type":"ContainerStarted","Data":"7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4"}
Oct 30 17:00:01.188685 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.188602    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77" event={"ID":"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51","Type":"ContainerStarted","Data":"fa3016e73ce60650163c291d0c61189da009d2edb7d465478e78a971482d3859"}
Oct 30 17:00:01.188685 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.188628    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 17:00:01.226294 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.226240    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77" podStartSLOduration=2.226223416 podStartE2EDuration="2.226223416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:01.22346648 +0000 UTC m=+2231.077344746" watchObservedRunningTime="2026-10-30 17:00:01.226223416 +0000 UTC m=+2231.080101681"
Oct 30 17:00:01.995276 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:01.990639    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"]
Oct 30 17:00:02.197011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:02.196964    4497 generic.go:356] "Generic (PLEG): container finished" podID="e2a0c469-e42e-40c0-8714-c1771a009900" containerID="a34b59b2ceeccb7f36380ac0d3b00f4be4910671a16bece0b6a7605858001800" exitCode=60
Oct 30 17:00:02.197205 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:02.197047    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr" event={"ID":"e2a0c469-e42e-40c0-8714-c1771a009900","Type":"ContainerDied","Data":"a34b59b2ceeccb7f36380ac0d3b00f4be4910671a16bece0b6a7605858001800"}
Oct 30 17:00:02.197476 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:02.197449    4497 scope.go:122] "RemoveContainer" containerID="a34b59b2ceeccb7f36380ac0d3b00f4be4910671a16bece0b6a7605858001800"
Oct 30 17:00:02.199092 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:02.199067    4497 generic.go:356] "Generic (PLEG): container finished" podID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerID="5c2e5903bec5ffc432ae334b7546e2a90a4e282127d61e80ebce9a5eceda3bd3" exitCode=60
Oct 30 17:00:02.199470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:02.199448    4497 scope.go:122] "RemoveContainer" containerID="5c2e5903bec5ffc432ae334b7546e2a90a4e282127d61e80ebce9a5eceda3bd3"
Oct 30 17:00:02.199758 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:02.199737    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6" event={"ID":"97cecf8a-82a6-4a3c-9fd7-e632506c2b45","Type":"ContainerDied","Data":"5c2e5903bec5ffc432ae334b7546e2a90a4e282127d61e80ebce9a5eceda3bd3"}
Oct 30 17:00:03.207784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:03.207756    4497 generic.go:356] "Generic (PLEG): container finished" podID="e2a0c469-e42e-40c0-8714-c1771a009900" containerID="8ac7cf5b5f6b5aad2a4ac5e4132126ff4bb08bc480a316430c51b803f9f0c511" exitCode=60
Oct 30 17:00:03.208203 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:03.207835    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr" event={"ID":"e2a0c469-e42e-40c0-8714-c1771a009900","Type":"ContainerDied","Data":"8ac7cf5b5f6b5aad2a4ac5e4132126ff4bb08bc480a316430c51b803f9f0c511"}
Oct 30 17:00:03.208203 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:03.207889    4497 scope.go:122] "RemoveContainer" containerID="a34b59b2ceeccb7f36380ac0d3b00f4be4910671a16bece0b6a7605858001800"
Oct 30 17:00:03.208305 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:03.208218    4497 scope.go:122] "RemoveContainer" containerID="8ac7cf5b5f6b5aad2a4ac5e4132126ff4bb08bc480a316430c51b803f9f0c511"
Oct 30 17:00:03.208576 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:03.208548    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr_odh-ai-gateway-infra(e2a0c469-e42e-40c0-8714-c1771a009900)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr" podUID="e2a0c469-e42e-40c0-8714-c1771a009900"
Oct 30 17:00:03.210237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:03.210218    4497 generic.go:356] "Generic (PLEG): container finished" podID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerID="de2f3e7c10db1072854a00e8e4ebee5f3ca57055a661c7ddf738b7dc51833c9b" exitCode=60
Oct 30 17:00:03.210362 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:03.210254    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6" event={"ID":"97cecf8a-82a6-4a3c-9fd7-e632506c2b45","Type":"ContainerDied","Data":"de2f3e7c10db1072854a00e8e4ebee5f3ca57055a661c7ddf738b7dc51833c9b"}
Oct 30 17:00:03.210513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:03.210496    4497 scope.go:122] "RemoveContainer" containerID="de2f3e7c10db1072854a00e8e4ebee5f3ca57055a661c7ddf738b7dc51833c9b"
Oct 30 17:00:03.210730 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:03.210710    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6_odh-ai-gateway-infra(97cecf8a-82a6-4a3c-9fd7-e632506c2b45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45"
Oct 30 17:00:03.297042 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:03.297019    4497 scope.go:122] "RemoveContainer" containerID="5c2e5903bec5ffc432ae334b7546e2a90a4e282127d61e80ebce9a5eceda3bd3"
Oct 30 17:00:04.044991 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.044959    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"]
Oct 30 17:00:04.050875 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.050851    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.066276 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.066249    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f94fb5-openshift-default-dockercfg-9vtqh\""
Oct 30 17:00:04.078088 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.078061    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"]
Oct 30 17:00:04.119362 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119332    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639dc55b-adf9-4a59-afc0-29f509b002b7-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.119536 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119400    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.119536 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119497    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntm9b\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-kube-api-access-ntm9b\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.119631 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119557    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.119676 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119646    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.119713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119684    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.119803 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119787    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.119848 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119822    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.119887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.119861    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.221062    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.221562    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.221845    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.221901    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.222022    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639dc55b-adf9-4a59-afc0-29f509b002b7-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.222119    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.222150    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntm9b\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-kube-api-access-ntm9b\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.222213    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.222284    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.222336    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.222911    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639dc55b-adf9-4a59-afc0-29f509b002b7-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.223165    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.224114    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.225548    4497 scope.go:122] "RemoveContainer" containerID="8ac7cf5b5f6b5aad2a4ac5e4132126ff4bb08bc480a316430c51b803f9f0c511"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:04.225834    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr_odh-ai-gateway-infra(e2a0c469-e42e-40c0-8714-c1771a009900)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr" podUID="e2a0c469-e42e-40c0-8714-c1771a009900"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.226365    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.226516    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.239224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.228689    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.258329 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.257867    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.274151 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.272389    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntm9b\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-kube-api-access-ntm9b\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.281382 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.281354    4497 scope.go:122] "RemoveContainer" containerID="de2f3e7c10db1072854a00e8e4ebee5f3ca57055a661c7ddf738b7dc51833c9b"
Oct 30 17:00:04.282054 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:04.282024    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6_odh-ai-gateway-infra(97cecf8a-82a6-4a3c-9fd7-e632506c2b45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45"
Oct 30 17:00:04.366614 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.366523    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:04.688732 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.688710    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"]
Oct 30 17:00:04.691452 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:04.691427    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639dc55b_adf9_4a59_afc0_29f509b002b7.slice/crio-20699a5f614bae27341d6132a2548ea5233e92c1780af65ee1af9f06d72490d1 WatchSource:0}: Error finding container 20699a5f614bae27341d6132a2548ea5233e92c1780af65ee1af9f06d72490d1: Status 404 returned error can't find the container with id 20699a5f614bae27341d6132a2548ea5233e92c1780af65ee1af9f06d72490d1
Oct 30 17:00:04.694407 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.694372    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:04.694501 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.694439    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:04.694501 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:04.694469    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:05.288480 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:05.288444    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" event={"ID":"639dc55b-adf9-4a59-afc0-29f509b002b7","Type":"ContainerStarted","Data":"be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00"}
Oct 30 17:00:05.288863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:05.288504    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" event={"ID":"639dc55b-adf9-4a59-afc0-29f509b002b7","Type":"ContainerStarted","Data":"20699a5f614bae27341d6132a2548ea5233e92c1780af65ee1af9f06d72490d1"}
Oct 30 17:00:05.346710 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:05.346647    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" podStartSLOduration=1.346632341 podStartE2EDuration="1.346632341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:05.334470632 +0000 UTC m=+2235.188348897" watchObservedRunningTime="2026-10-30 17:00:05.346632341 +0000 UTC m=+2235.200510599"
Oct 30 17:00:05.367560 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:05.367528    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:05.373394 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:05.373369    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:06.333307 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.332314    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:06.333307 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.332405    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:00:06.468741 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.468708    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"]
Oct 30 17:00:06.472918 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.472900    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.485479 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.485458    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:00:06.493241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.493215    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"]
Oct 30 17:00:06.560519 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.560487    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.560678 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.560534    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.560678 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.560606    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5r4g\" (UniqueName: \"kubernetes.io/projected/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-kube-api-access-h5r4g\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.661788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.661702    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.661788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.661757    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.662011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.661799    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5r4g\" (UniqueName: \"kubernetes.io/projected/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-kube-api-access-h5r4g\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.662415 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.662397    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.667482 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.667431    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.684690 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.684662    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5r4g\" (UniqueName: \"kubernetes.io/projected/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-kube-api-access-h5r4g\") pod \"maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:06.788089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:06.788052    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:07.092653 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:07.092623    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"]
Oct 30 17:00:07.096274 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:07.096242    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a71c0d7_3d56_47d6_ab00_dc1f2f2ff684.slice/crio-ca997e15dfa29481f4490051fa8020a324f1b02dea298acbd7b46b71b42559c6 WatchSource:0}: Error finding container ca997e15dfa29481f4490051fa8020a324f1b02dea298acbd7b46b71b42559c6: Status 404 returned error can't find the container with id ca997e15dfa29481f4490051fa8020a324f1b02dea298acbd7b46b71b42559c6
Oct 30 17:00:07.331155 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:07.331120    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm" event={"ID":"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684","Type":"ContainerStarted","Data":"ca997e15dfa29481f4490051fa8020a324f1b02dea298acbd7b46b71b42559c6"}
Oct 30 17:00:08.338497 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:08.338462    4497 generic.go:356] "Generic (PLEG): container finished" podID="5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684" containerID="18c879a7a59a595c62e894c4637deb0bed1a8daa2d10508e0d6c936fa5c823b5" exitCode=0
Oct 30 17:00:08.338935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:08.338549    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm" event={"ID":"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684","Type":"ContainerDied","Data":"18c879a7a59a595c62e894c4637deb0bed1a8daa2d10508e0d6c936fa5c823b5"}
Oct 30 17:00:09.501392 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.501366    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:09.603780 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.603687    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-xks-ca\") pod \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") "
Oct 30 17:00:09.603959 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.603799    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-openshift-ca\") pod \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") "
Oct 30 17:00:09.603959 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.603901    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-kube-api-access-h5r4g\" (UniqueName: \"kubernetes.io/projected/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-kube-api-access-h5r4g\") pod \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\" (UID: \"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684\") "
Oct 30 17:00:09.604163 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.604143    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-openshift-ca" pod "5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684" (UID: "5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:09.604533 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.604516    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-openshift-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:09.605895 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.605861    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-kube-api-access-h5r4g" pod "5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684" (UID: "5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684"). InnerVolumeSpecName "kube-api-access-h5r4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:09.605895 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.605867    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-xks-ca" pod "5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684" (UID: "5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:00:09.705834 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.705801    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-maas-api-xks-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:09.705834 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:09.705831    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5r4g\" (UniqueName: \"kubernetes.io/projected/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684-kube-api-access-h5r4g\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:10.352425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:10.352388    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"
Oct 30 17:00:10.352618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:10.352430    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm" event={"ID":"5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684","Type":"ContainerDied","Data":"ca997e15dfa29481f4490051fa8020a324f1b02dea298acbd7b46b71b42559c6"}
Oct 30 17:00:10.352618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:10.352487    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca997e15dfa29481f4490051fa8020a324f1b02dea298acbd7b46b71b42559c6"
Oct 30 17:00:12.203164 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:12.203132    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 17:00:16.293334 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:16.293300    4497 scope.go:122] "RemoveContainer" containerID="de2f3e7c10db1072854a00e8e4ebee5f3ca57055a661c7ddf738b7dc51833c9b"
Oct 30 17:00:17.016430 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.016394    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp" podUID="a036fa80-7f42-4286-9d47-4e3d250dcb65" containerName="main" containerID="cri-o://d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d" gracePeriod=45
Oct 30 17:00:17.293789 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.293217    4497 scope.go:122] "RemoveContainer" containerID="8ac7cf5b5f6b5aad2a4ac5e4132126ff4bb08bc480a316430c51b803f9f0c511"
Oct 30 17:00:17.397042 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.397012    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 17:00:17.409250 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.409214    4497 generic.go:356] "Generic (PLEG): container finished" podID="a036fa80-7f42-4286-9d47-4e3d250dcb65" containerID="d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d" exitCode=0
Oct 30 17:00:17.409387 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.409253    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"
Oct 30 17:00:17.409387 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.409297    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp" event={"ID":"a036fa80-7f42-4286-9d47-4e3d250dcb65","Type":"ContainerDied","Data":"d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d"}
Oct 30 17:00:17.409387 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.409367    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp" event={"ID":"a036fa80-7f42-4286-9d47-4e3d250dcb65","Type":"ContainerDied","Data":"ff5a347cf8989132e94d021beb880fcb5c8bc408723d3e5b1fb1d918ea590900"}
Oct 30 17:00:17.409526 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.409392    4497 scope.go:122] "RemoveContainer" containerID="d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d"
Oct 30 17:00:17.411641 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.411619    4497 generic.go:356] "Generic (PLEG): container finished" podID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerID="c2ffe29b70f509cbb4b540f410c846e5700ad7b7470c3b5494faafe3c9c8b097" exitCode=60
Oct 30 17:00:17.411761 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.411698    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6" event={"ID":"97cecf8a-82a6-4a3c-9fd7-e632506c2b45","Type":"ContainerDied","Data":"c2ffe29b70f509cbb4b540f410c846e5700ad7b7470c3b5494faafe3c9c8b097"}
Oct 30 17:00:17.412168 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.412152    4497 scope.go:122] "RemoveContainer" containerID="c2ffe29b70f509cbb4b540f410c846e5700ad7b7470c3b5494faafe3c9c8b097"
Oct 30 17:00:17.412534 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:17.412514    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6_odh-ai-gateway-infra(97cecf8a-82a6-4a3c-9fd7-e632506c2b45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45"
Oct 30 17:00:17.420269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.420249    4497 scope.go:122] "RemoveContainer" containerID="d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d"
Oct 30 17:00:17.420799 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:17.420770    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d\": container with ID starting with d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d not found: ID does not exist" containerID="d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d"
Oct 30 17:00:17.420923 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.420894    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d"} err="failed to get container status \"d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d\": rpc error: code = NotFound desc = could not find container \"d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d\": container with ID starting with d9d155b531c9cc3e137b2d6ce862f3f2ce017f309b6946e4f4f3f91cf5a28f3d not found: ID does not exist"
Oct 30 17:00:17.420923 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.420919    4497 scope.go:122] "RemoveContainer" containerID="de2f3e7c10db1072854a00e8e4ebee5f3ca57055a661c7ddf738b7dc51833c9b"
Oct 30 17:00:17.521208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521151    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-tmp-dir\") pod \"a036fa80-7f42-4286-9d47-4e3d250dcb65\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") "
Oct 30 17:00:17.521384 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521222    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a036fa80-7f42-4286-9d47-4e3d250dcb65-tls-certs\" (UniqueName: \"kubernetes.io/secret/a036fa80-7f42-4286-9d47-4e3d250dcb65-tls-certs\") pod \"a036fa80-7f42-4286-9d47-4e3d250dcb65\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") "
Oct 30 17:00:17.521434 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521383    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-model-cache\") pod \"a036fa80-7f42-4286-9d47-4e3d250dcb65\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") "
Oct 30 17:00:17.521434 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521403    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-dshm\") pod \"a036fa80-7f42-4286-9d47-4e3d250dcb65\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") "
Oct 30 17:00:17.521434 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521432    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a036fa80-7f42-4286-9d47-4e3d250dcb65-kube-api-access-4rpcb\" (UniqueName: \"kubernetes.io/projected/a036fa80-7f42-4286-9d47-4e3d250dcb65-kube-api-access-4rpcb\") pod \"a036fa80-7f42-4286-9d47-4e3d250dcb65\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") "
Oct 30 17:00:17.521605 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521473    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-home\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-home\") pod \"a036fa80-7f42-4286-9d47-4e3d250dcb65\" (UID: \"a036fa80-7f42-4286-9d47-4e3d250dcb65\") "
Oct 30 17:00:17.521605 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521486    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-tmp-dir" pod "a036fa80-7f42-4286-9d47-4e3d250dcb65" (UID: "a036fa80-7f42-4286-9d47-4e3d250dcb65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:17.521717 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521679    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-model-cache" pod "a036fa80-7f42-4286-9d47-4e3d250dcb65" (UID: "a036fa80-7f42-4286-9d47-4e3d250dcb65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:17.521795 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.521776    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-home" pod "a036fa80-7f42-4286-9d47-4e3d250dcb65" (UID: "a036fa80-7f42-4286-9d47-4e3d250dcb65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:17.522275 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.522253    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:17.522402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.522278    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:17.522402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.522292    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:17.524166 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.524138    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-dshm" pod "a036fa80-7f42-4286-9d47-4e3d250dcb65" (UID: "a036fa80-7f42-4286-9d47-4e3d250dcb65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:17.524274 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.524195    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a036fa80-7f42-4286-9d47-4e3d250dcb65-tls-certs" pod "a036fa80-7f42-4286-9d47-4e3d250dcb65" (UID: "a036fa80-7f42-4286-9d47-4e3d250dcb65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:00:17.524322 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.524262    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a036fa80-7f42-4286-9d47-4e3d250dcb65-kube-api-access-4rpcb" pod "a036fa80-7f42-4286-9d47-4e3d250dcb65" (UID: "a036fa80-7f42-4286-9d47-4e3d250dcb65"). InnerVolumeSpecName "kube-api-access-4rpcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:17.624813 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.624780    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a036fa80-7f42-4286-9d47-4e3d250dcb65-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:17.624813 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.624813    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a036fa80-7f42-4286-9d47-4e3d250dcb65-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:17.625035 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.624827    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rpcb\" (UniqueName: \"kubernetes.io/projected/a036fa80-7f42-4286-9d47-4e3d250dcb65-kube-api-access-4rpcb\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:17.803208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.802285    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"]
Oct 30 17:00:17.803208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:17.802767    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3ca339/auth-test-model-3ca339-kserve-64885cd64f-pt5fp"]
Oct 30 17:00:18.301126 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.301091    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a036fa80-7f42-4286-9d47-4e3d250dcb65" path="/var/lib/kubelet/pods/a036fa80-7f42-4286-9d47-4e3d250dcb65/volumes"
Oct 30 17:00:18.421934 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.421842    4497 generic.go:356] "Generic (PLEG): container finished" podID="e2a0c469-e42e-40c0-8714-c1771a009900" containerID="ef71df62edf3e8ada88cbaac7db63d033fe78dcf4bba3d5c8b357b2f2b43ea3a" exitCode=60
Oct 30 17:00:18.421934 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.421913    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr" event={"ID":"e2a0c469-e42e-40c0-8714-c1771a009900","Type":"ContainerDied","Data":"ef71df62edf3e8ada88cbaac7db63d033fe78dcf4bba3d5c8b357b2f2b43ea3a"}
Oct 30 17:00:18.422144 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.421970    4497 scope.go:122] "RemoveContainer" containerID="8ac7cf5b5f6b5aad2a4ac5e4132126ff4bb08bc480a316430c51b803f9f0c511"
Oct 30 17:00:18.422456 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.422435    4497 scope.go:122] "RemoveContainer" containerID="ef71df62edf3e8ada88cbaac7db63d033fe78dcf4bba3d5c8b357b2f2b43ea3a"
Oct 30 17:00:18.423276 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:18.422802    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr_odh-ai-gateway-infra(e2a0c469-e42e-40c0-8714-c1771a009900)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr" podUID="e2a0c469-e42e-40c0-8714-c1771a009900"
Oct 30 17:00:18.483133 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.483087    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"]
Oct 30 17:00:18.659957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.659928    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:18.742846 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.742759    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-maas-api-service-ca\") pod \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\" (UID: \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\") "
Oct 30 17:00:18.742846 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.742807    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-kube-api-access-g4k5w\" (UniqueName: \"kubernetes.io/projected/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-kube-api-access-g4k5w\") pod \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\" (UID: \"97cecf8a-82a6-4a3c-9fd7-e632506c2b45\") "
Oct 30 17:00:18.743134 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.743114    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-maas-api-service-ca" pod "97cecf8a-82a6-4a3c-9fd7-e632506c2b45" (UID: "97cecf8a-82a6-4a3c-9fd7-e632506c2b45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:18.743429 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.743406    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:18.747369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.747338    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-kube-api-access-g4k5w" pod "97cecf8a-82a6-4a3c-9fd7-e632506c2b45" (UID: "97cecf8a-82a6-4a3c-9fd7-e632506c2b45"). InnerVolumeSpecName "kube-api-access-g4k5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:18.845180 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:18.845141    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4k5w\" (UniqueName: \"kubernetes.io/projected/97cecf8a-82a6-4a3c-9fd7-e632506c2b45-kube-api-access-g4k5w\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:19.436222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.436150    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"
Oct 30 17:00:19.436222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.436170    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6" event={"ID":"97cecf8a-82a6-4a3c-9fd7-e632506c2b45","Type":"ContainerDied","Data":"30e2e34f572bbf9c87cc5ebc2625922dcf2712b755d72466bc92bf52eab2e262"}
Oct 30 17:00:19.436777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.436246    4497 scope.go:122] "RemoveContainer" containerID="c2ffe29b70f509cbb4b540f410c846e5700ad7b7470c3b5494faafe3c9c8b097"
Oct 30 17:00:19.473115 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.473084    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"]
Oct 30 17:00:19.480411 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.480386    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"]
Oct 30 17:00:19.488664 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.488637    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889660-hffz6"]
Oct 30 17:00:19.625648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.625616    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:19.757257 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.757226    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a0c469-e42e-40c0-8714-c1771a009900-kube-api-access-2blhd\" (UniqueName: \"kubernetes.io/projected/e2a0c469-e42e-40c0-8714-c1771a009900-kube-api-access-2blhd\") pod \"e2a0c469-e42e-40c0-8714-c1771a009900\" (UID: \"e2a0c469-e42e-40c0-8714-c1771a009900\") "
Oct 30 17:00:19.757452 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.757392    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a0c469-e42e-40c0-8714-c1771a009900-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0c469-e42e-40c0-8714-c1771a009900-maas-api-service-ca\") pod \"e2a0c469-e42e-40c0-8714-c1771a009900\" (UID: \"e2a0c469-e42e-40c0-8714-c1771a009900\") "
Oct 30 17:00:19.757770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.757745    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a0c469-e42e-40c0-8714-c1771a009900-maas-api-service-ca" pod "e2a0c469-e42e-40c0-8714-c1771a009900" (UID: "e2a0c469-e42e-40c0-8714-c1771a009900"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:19.758077 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.758059    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a0c469-e42e-40c0-8714-c1771a009900-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:19.759496 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.759462    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a0c469-e42e-40c0-8714-c1771a009900-kube-api-access-2blhd" pod "e2a0c469-e42e-40c0-8714-c1771a009900" (UID: "e2a0c469-e42e-40c0-8714-c1771a009900"). InnerVolumeSpecName "kube-api-access-2blhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:19.859347 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.859312    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blhd\" (UniqueName: \"kubernetes.io/projected/e2a0c469-e42e-40c0-8714-c1771a009900-kube-api-access-2blhd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:19.980690 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.980655    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"]
Oct 30 17:00:19.980961 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:19.980931    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m" podUID="eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" containerName="maas-api" containerID="cri-o://037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d" gracePeriod=30
Oct 30 17:00:20.303413 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.303382    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" path="/var/lib/kubelet/pods/97cecf8a-82a6-4a3c-9fd7-e632506c2b45/volumes"
Oct 30 17:00:20.304218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.304173    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 17:00:20.448173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.448080    4497 generic.go:356] "Generic (PLEG): container finished" podID="eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" containerID="037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d" exitCode=0
Oct 30 17:00:20.448173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.448141    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"
Oct 30 17:00:20.448654 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.448167    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m" event={"ID":"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07","Type":"ContainerDied","Data":"037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d"}
Oct 30 17:00:20.448654 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.448252    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m" event={"ID":"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07","Type":"ContainerDied","Data":"27b10c730ed0f624327cfdfdba74220b12fc0a14c0a1f352d22c15e1551764ba"}
Oct 30 17:00:20.448654 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.448287    4497 scope.go:122] "RemoveContainer" containerID="037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d"
Oct 30 17:00:20.450466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.450369    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"
Oct 30 17:00:20.450466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.450386    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr" event={"ID":"e2a0c469-e42e-40c0-8714-c1771a009900","Type":"ContainerDied","Data":"800aa467405cbaf68ca9a34b58f0961fb1d7ee94966f48fcd3b8cd312dbfb2bb"}
Oct 30 17:00:20.459388 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.459367    4497 scope.go:122] "RemoveContainer" containerID="037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d"
Oct 30 17:00:20.459661 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:20.459643    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d\": container with ID starting with 037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d not found: ID does not exist" containerID="037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d"
Oct 30 17:00:20.459751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.459671    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d"} err="failed to get container status \"037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d\": rpc error: code = NotFound desc = could not find container \"037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d\": container with ID starting with 037bd9ec6cdb88b09834a85f34f762bee83a613df41ba456e0e687ae150ac65d not found: ID does not exist"
Oct 30 17:00:20.459751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.459693    4497 scope.go:122] "RemoveContainer" containerID="ef71df62edf3e8ada88cbaac7db63d033fe78dcf4bba3d5c8b357b2f2b43ea3a"
Oct 30 17:00:20.523237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.472532    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-openshift-service-ca\") pod \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") "
Oct 30 17:00:20.523237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.472716    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-kube-api-access-92cqz\" (UniqueName: \"kubernetes.io/projected/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-kube-api-access-92cqz\") pod \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") "
Oct 30 17:00:20.523237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.472859    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-maas-api-tls\") pod \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") "
Oct 30 17:00:20.523237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.472963    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-metrics-certs\") pod \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\" (UID: \"eeb39e43-ec8d-4b5f-a850-0e49c32c3a07\") "
Oct 30 17:00:20.523237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.474301    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-openshift-service-ca" pod "eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" (UID: "eeb39e43-ec8d-4b5f-a850-0e49c32c3a07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:20.523237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.477753    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-kube-api-access-92cqz" pod "eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" (UID: "eeb39e43-ec8d-4b5f-a850-0e49c32c3a07"). InnerVolumeSpecName "kube-api-access-92cqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:20.523237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.492601    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-metrics-certs" pod "eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" (UID: "eeb39e43-ec8d-4b5f-a850-0e49c32c3a07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:00:20.549509 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.541258    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"]
Oct 30 17:00:20.549509 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.545018    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889660-v68jr"]
Oct 30 17:00:20.554860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.554793    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-maas-api-tls" pod "eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" (UID: "eeb39e43-ec8d-4b5f-a850-0e49c32c3a07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:00:20.575160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.575116    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92cqz\" (UniqueName: \"kubernetes.io/projected/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-kube-api-access-92cqz\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:20.575160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.575158    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:20.575160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.575167    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:20.575433 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.575175    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:20.775409 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.775270    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"]
Oct 30 17:00:20.777327 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.777209    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-18fb02-xtgnm"]
Oct 30 17:00:20.814068 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.814028    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"]
Oct 30 17:00:20.817378 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:20.817350    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-18fb02-5698758f64-vmb2m"]
Oct 30 17:00:21.917820 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:21.917780    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:00:22.300258 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:22.300219    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684" path="/var/lib/kubelet/pods/5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684/volumes"
Oct 30 17:00:22.300624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:22.300610    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" path="/var/lib/kubelet/pods/e2a0c469-e42e-40c0-8714-c1771a009900/volumes"
Oct 30 17:00:22.301008 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:22.300996    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" path="/var/lib/kubelet/pods/eeb39e43-ec8d-4b5f-a850-0e49c32c3a07/volumes"
Oct 30 17:00:28.011682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.011650    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"]
Oct 30 17:00:28.012412 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012389    4497 cpu_manager.go:414] "containerMap: removing container" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:28.012412 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012411    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012428    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012436    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012452    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a036fa80-7f42-4286-9d47-4e3d250dcb65" containerName="main"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012459    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a036fa80-7f42-4286-9d47-4e3d250dcb65" containerName="main"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012475    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012482    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012515    4497 cpu_manager.go:414] "containerMap: removing container" podUID="eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" containerName="maas-api"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012522    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" containerName="maas-api"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012539    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684" containerName="revoke-keys"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012547    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684" containerName="revoke-keys"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012563    4497 cpu_manager.go:414] "containerMap: removing container" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:28.012625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012571    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:28.013260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012770    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:28.013260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012784    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:28.013260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012794    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:28.013260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012808    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a036fa80-7f42-4286-9d47-4e3d250dcb65" containerName="main"
Oct 30 17:00:28.013260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012817    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:28.013260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012830    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb39e43-ec8d-4b5f-a850-0e49c32c3a07" containerName="maas-api"
Oct 30 17:00:28.013260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012838    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a71c0d7-3d56-47d6-ab00-dc1f2f2ff684" containerName="revoke-keys"
Oct 30 17:00:28.013260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.012849    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:28.018683 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.018658    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.031976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.031949    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2589b5-openshift-default-dockercfg-r6f89\""
Oct 30 17:00:28.050645 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.050612    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"]
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187223    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhq9r\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-kube-api-access-nhq9r\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187300    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-certs\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187325    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-data\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187391    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-credential-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187549    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187601    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-envoy\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187623    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istiod-ca-cert\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187663    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-podinfo\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.188214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.187692    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-token\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289249 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289152    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289249 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289243    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-envoy\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289263    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istiod-ca-cert\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289284    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-podinfo\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289302    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-token\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289384    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhq9r\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-kube-api-access-nhq9r\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289429    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-certs\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289453    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-data\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289811 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289513    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-credential-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289811 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289614    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.289811 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289766    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-certs\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.290016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.289978    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-data\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.290077 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.290015    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-credential-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.290168 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.290152    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istiod-ca-cert\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.291863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.291840    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-envoy\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.291988 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.291966    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-podinfo\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.319147 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.319108    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-token\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.319322 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.319293    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhq9r\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-kube-api-access-nhq9r\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.334202 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.334164    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:28.672017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.671941    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"]
Oct 30 17:00:28.677243 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:28.677215    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95cc0cc_dbf7_4fc8_8fb7_9867a69b97e0.slice/crio-70c3f7c5b195898ff92b08568c18f86ab294a2a88b9c39dc6aecd2a1c18fae8b WatchSource:0}: Error finding container 70c3f7c5b195898ff92b08568c18f86ab294a2a88b9c39dc6aecd2a1c18fae8b: Status 404 returned error can't find the container with id 70c3f7c5b195898ff92b08568c18f86ab294a2a88b9c39dc6aecd2a1c18fae8b
Oct 30 17:00:28.680136 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.680098    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:28.680279 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.680205    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:28.680279 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:28.680255    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:29.126130 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.126092    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7440-3bd314"]
Oct 30 17:00:29.126740 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.126725    4497 cpu_manager.go:414] "containerMap: removing container" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:29.126786 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.126741    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:29.126786 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.126777    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:29.126786 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.126783    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a0c469-e42e-40c0-8714-c1771a009900" containerName="cleanup"
Oct 30 17:00:29.126880 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.126874    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cecf8a-82a6-4a3c-9fd7-e632506c2b45" containerName="cleanup"
Oct 30 17:00:29.130312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.130292    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:29.142386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.142360    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qn5rq\""
Oct 30 17:00:29.149295 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.149271    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7440-3bd314"]
Oct 30 17:00:29.312575 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.312540    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvdh\" (UniqueName: \"kubernetes.io/projected/c1a98a12-0704-49d1-b4bb-2247e815909f-kube-api-access-vtvdh\") pod \"test-curl-7440-3bd314\" (UID: \"c1a98a12-0704-49d1-b4bb-2247e815909f\") " pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:29.414559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.414467    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvdh\" (UniqueName: \"kubernetes.io/projected/c1a98a12-0704-49d1-b4bb-2247e815909f-kube-api-access-vtvdh\") pod \"test-curl-7440-3bd314\" (UID: \"c1a98a12-0704-49d1-b4bb-2247e815909f\") " pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:29.436053 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.436024    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvdh\" (UniqueName: \"kubernetes.io/projected/c1a98a12-0704-49d1-b4bb-2247e815909f-kube-api-access-vtvdh\") pod \"test-curl-7440-3bd314\" (UID: \"c1a98a12-0704-49d1-b4bb-2247e815909f\") " pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:29.442967 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.442939    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:29.535867 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.535833    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" event={"ID":"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0","Type":"ContainerStarted","Data":"ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970"}
Oct 30 17:00:29.536071 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.535889    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" event={"ID":"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0","Type":"ContainerStarted","Data":"70c3f7c5b195898ff92b08568c18f86ab294a2a88b9c39dc6aecd2a1c18fae8b"}
Oct 30 17:00:29.576298 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.576243    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" podStartSLOduration=2.5762211280000002 podStartE2EDuration="2.576221128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:29.575532711 +0000 UTC m=+2259.429410977" watchObservedRunningTime="2026-10-30 17:00:29.576221128 +0000 UTC m=+2259.430099393"
Oct 30 17:00:29.759177 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:29.759144    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7440-3bd314"]
Oct 30 17:00:29.762627 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:29.762596    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1a98a12_0704_49d1_b4bb_2247e815909f.slice/crio-44114531f7f2f548def300bae1f48420ceb8cb16f2f9903c1ffd55ad8895f8ea WatchSource:0}: Error finding container 44114531f7f2f548def300bae1f48420ceb8cb16f2f9903c1ffd55ad8895f8ea: Status 404 returned error can't find the container with id 44114531f7f2f548def300bae1f48420ceb8cb16f2f9903c1ffd55ad8895f8ea
Oct 30 17:00:30.334764 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:30.334730    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:30.340747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:30.340719    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:30.543643 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:30.543602    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7440-3bd314" event={"ID":"c1a98a12-0704-49d1-b4bb-2247e815909f","Type":"ContainerStarted","Data":"55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6"}
Oct 30 17:00:30.543837 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:30.543655    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7440-3bd314" event={"ID":"c1a98a12-0704-49d1-b4bb-2247e815909f","Type":"ContainerStarted","Data":"44114531f7f2f548def300bae1f48420ceb8cb16f2f9903c1ffd55ad8895f8ea"}
Oct 30 17:00:30.544049 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:30.544031    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:30.544990 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:30.544970    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:00:30.580056 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:30.580006    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7440-3bd314" podStartSLOduration=1.5799895369999999 podStartE2EDuration="1.579989537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:30.579047898 +0000 UTC m=+2260.432926165" watchObservedRunningTime="2026-10-30 17:00:30.579989537 +0000 UTC m=+2260.433867805"
Oct 30 17:00:31.473269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:31.473236    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7440-3bd314"]
Oct 30 17:00:31.476462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:31.476431    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7440-3bd314"]
Oct 30 17:00:32.556396 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:32.556352    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7440-3bd314" podUID="c1a98a12-0704-49d1-b4bb-2247e815909f" containerName="test-curl-7440-3bd314" containerID="cri-o://55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6" gracePeriod=2
Oct 30 17:00:33.628757 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.628723    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"]
Oct 30 17:00:33.629754 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.629731    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c1a98a12-0704-49d1-b4bb-2247e815909f" containerName="test-curl-7440-3bd314"
Oct 30 17:00:33.629754 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.629754    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a98a12-0704-49d1-b4bb-2247e815909f" containerName="test-curl-7440-3bd314"
Oct 30 17:00:33.629948 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.629934    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a98a12-0704-49d1-b4bb-2247e815909f" containerName="test-curl-7440-3bd314"
Oct 30 17:00:33.635226 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.635204    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.656919 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.656887    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"]
Oct 30 17:00:33.660698 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.660672    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3106d6-openshift-default-dockercfg-hpqnn\""
Oct 30 17:00:33.672965 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.672932    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.673135 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.672979    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-data\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.673135 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.673049    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-credential-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.673264 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.673169    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9hn\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-kube-api-access-vc9hn\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.673264 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.673231    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-token\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.673264 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.673255    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istiod-ca-cert\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.673412 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.673310    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-certs\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.673412 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.673330    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-podinfo\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.673412 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.673347    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-envoy\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.774561 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774471    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-token\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.774561 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774513    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istiod-ca-cert\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.774817 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774578    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-certs\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.774817 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774607    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-podinfo\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.774817 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774633    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-envoy\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.774817 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774789    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.775029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774830    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-data\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.775029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774864    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-credential-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.775029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774951    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9hn\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-kube-api-access-vc9hn\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.775029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.774983    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-certs\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.775267 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.775207    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.775358 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.775332    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-credential-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.775418 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.775396    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istiod-ca-cert\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.775554 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.775535    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-data\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.777005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.776977    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-envoy\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.777240 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.777222    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-podinfo\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.802976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.802931    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9hn\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-kube-api-access-vc9hn\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.803473 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.803448    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-token\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:33.957210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:33.953470    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:34.378302 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.378278    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"]
Oct 30 17:00:34.381200 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:34.381160    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e7bdc0_1950_4c66_96d6_30caa8f8c2c0.slice/crio-893432a4f72365aa94fd2694fec66a4785f7082a2b8da087dc581c7338e7c354 WatchSource:0}: Error finding container 893432a4f72365aa94fd2694fec66a4785f7082a2b8da087dc581c7338e7c354: Status 404 returned error can't find the container with id 893432a4f72365aa94fd2694fec66a4785f7082a2b8da087dc581c7338e7c354
Oct 30 17:00:34.383952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.383914    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:34.384037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.383984    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:34.384037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.384016    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:34.572522 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.572491    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" event={"ID":"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0","Type":"ContainerStarted","Data":"19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15"}
Oct 30 17:00:34.572652 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.572543    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" event={"ID":"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0","Type":"ContainerStarted","Data":"893432a4f72365aa94fd2694fec66a4785f7082a2b8da087dc581c7338e7c354"}
Oct 30 17:00:34.618847 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.618761    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" podStartSLOduration=1.618740466 podStartE2EDuration="1.618740466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:34.614106439 +0000 UTC m=+2264.467984703" watchObservedRunningTime="2026-10-30 17:00:34.618740466 +0000 UTC m=+2264.472618734"
Oct 30 17:00:34.712325 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.712301    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:34.806616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.797735    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a98a12-0704-49d1-b4bb-2247e815909f-kube-api-access-vtvdh\" (UniqueName: \"kubernetes.io/projected/c1a98a12-0704-49d1-b4bb-2247e815909f-kube-api-access-vtvdh\") pod \"c1a98a12-0704-49d1-b4bb-2247e815909f\" (UID: \"c1a98a12-0704-49d1-b4bb-2247e815909f\") "
Oct 30 17:00:34.806616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.801909    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a98a12-0704-49d1-b4bb-2247e815909f-kube-api-access-vtvdh" pod "c1a98a12-0704-49d1-b4bb-2247e815909f" (UID: "c1a98a12-0704-49d1-b4bb-2247e815909f"). InnerVolumeSpecName "kube-api-access-vtvdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:34.900137 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.900107    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvdh\" (UniqueName: \"kubernetes.io/projected/c1a98a12-0704-49d1-b4bb-2247e815909f-kube-api-access-vtvdh\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:34.955237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:34.955137    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:35.579747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:35.579709    4497 generic.go:356] "Generic (PLEG): container finished" podID="c1a98a12-0704-49d1-b4bb-2247e815909f" containerID="55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6" exitCode=137
Oct 30 17:00:35.579747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:35.579750    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:35.579978 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:35.579811    4497 scope.go:122] "RemoveContainer" containerID="55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6"
Oct 30 17:00:35.589850 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:35.589826    4497 scope.go:122] "RemoveContainer" containerID="55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6"
Oct 30 17:00:35.590120 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:35.590101    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6\": container with ID starting with 55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6 not found: ID does not exist" containerID="55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6"
Oct 30 17:00:35.590176 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:35.590133    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6"} err="failed to get container status \"55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6\": rpc error: code = NotFound desc = could not find container \"55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6\": container with ID starting with 55c9af9c1493dccc30d2894e19ec468a484074f2f01b642748c032543e0152d6 not found: ID does not exist"
Oct 30 17:00:35.600040 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:35.600008    4497 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7440-3bd314\" is forbidden: User \"system:node:ip-10-0-136-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-74.ec2.internal' and this object" podUID="c1a98a12-0704-49d1-b4bb-2247e815909f" pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:35.961028 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:35.960951    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:35.972153 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:35.972120    4497 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7440-3bd314\" is forbidden: User \"system:node:ip-10-0-136-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-74.ec2.internal' and this object" podUID="c1a98a12-0704-49d1-b4bb-2247e815909f" pod="opendatahub/test-curl-7440-3bd314"
Oct 30 17:00:36.302680 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:36.302648    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a98a12-0704-49d1-b4bb-2247e815909f" path="/var/lib/kubelet/pods/c1a98a12-0704-49d1-b4bb-2247e815909f/volumes"
Oct 30 17:00:36.587247 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:36.587142    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:36.588102 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:36.588078    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:00:39.503976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.503938    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"]
Oct 30 17:00:39.509695 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.509676    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:39.527689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.527662    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"]
Oct 30 17:00:39.533121 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.533096    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-3106d6-dockercfg-xdf2v\""
Oct 30 17:00:39.559390 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.559356    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3106d6\""
Oct 30 17:00:39.647756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.647717    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qpb2\" (UniqueName: \"kubernetes.io/projected/62009120-1b30-4812-9e31-0aeaf65896a7-kube-api-access-6qpb2\") pod \"payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9\" (UID: \"62009120-1b30-4812-9e31-0aeaf65896a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:39.654086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.654052    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009120-1b30-4812-9e31-0aeaf65896a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9\" (UID: \"62009120-1b30-4812-9e31-0aeaf65896a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:39.755907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.755818    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qpb2\" (UniqueName: \"kubernetes.io/projected/62009120-1b30-4812-9e31-0aeaf65896a7-kube-api-access-6qpb2\") pod \"payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9\" (UID: \"62009120-1b30-4812-9e31-0aeaf65896a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:39.755907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.755863    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009120-1b30-4812-9e31-0aeaf65896a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9\" (UID: \"62009120-1b30-4812-9e31-0aeaf65896a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:39.756675 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.756653    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009120-1b30-4812-9e31-0aeaf65896a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9\" (UID: \"62009120-1b30-4812-9e31-0aeaf65896a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:39.777539 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.777507    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qpb2\" (UniqueName: \"kubernetes.io/projected/62009120-1b30-4812-9e31-0aeaf65896a7-kube-api-access-6qpb2\") pod \"payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9\" (UID: \"62009120-1b30-4812-9e31-0aeaf65896a7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:39.824468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:39.824430    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:40.135034 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:40.134954    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"]
Oct 30 17:00:40.138447 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:00:40.138418    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62009120_1b30_4812_9e31_0aeaf65896a7.slice/crio-8c5552b4acb48a5f2b4094a937156b0728377cb27ce711c201d91c44e9c2b507 WatchSource:0}: Error finding container 8c5552b4acb48a5f2b4094a937156b0728377cb27ce711c201d91c44e9c2b507: Status 404 returned error can't find the container with id 8c5552b4acb48a5f2b4094a937156b0728377cb27ce711c201d91c44e9c2b507
Oct 30 17:00:40.664090 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:40.664056    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9" event={"ID":"62009120-1b30-4812-9e31-0aeaf65896a7","Type":"ContainerStarted","Data":"d61b09e19c53fc8d26095df6e7349bf98be2a94f73e2d44999e5fddd6b556cbf"}
Oct 30 17:00:40.664511 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:40.664108    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9" event={"ID":"62009120-1b30-4812-9e31-0aeaf65896a7","Type":"ContainerStarted","Data":"8c5552b4acb48a5f2b4094a937156b0728377cb27ce711c201d91c44e9c2b507"}
Oct 30 17:00:40.664511 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:40.664137    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:40.699754 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:40.699704    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9" podStartSLOduration=1.699689375 podStartE2EDuration="1.699689375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:40.698998778 +0000 UTC m=+2270.552877060" watchObservedRunningTime="2026-10-30 17:00:40.699689375 +0000 UTC m=+2270.553567639"
Oct 30 17:00:46.702804 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:46.702767    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"]
Oct 30 17:00:46.703676 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:46.703597    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy" containerID="cri-o://f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4" gracePeriod=30
Oct 30 17:00:51.672952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:51.672870    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:00:51.768390 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:51.768356    4497 generic.go:356] "Generic (PLEG): container finished" podID="9905c187-904a-44ee-ab84-f6e767fc887b" containerID="33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94" exitCode=0
Oct 30 17:00:51.768565 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:51.768423    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" event={"ID":"9905c187-904a-44ee-ab84-f6e767fc887b","Type":"ContainerDied","Data":"33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94"}
Oct 30 17:00:51.768826 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:51.768813    4497 scope.go:122] "RemoveContainer" containerID="33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94"
Oct 30 17:00:51.770312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:51.770279    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:51.770441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:51.770349    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:51.770441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:51.770381    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:00:51.947286 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:51.945581    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:00:52.018682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.018651    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 17:00:52.174897 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.174861    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-certs\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175100 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175012    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-podinfo\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175100 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175056    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-credential-socket\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175277 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175103    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-token\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175277 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175221    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-data\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175277 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175258    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-socket\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175291    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-kube-api-access-9cwfq\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-kube-api-access-9cwfq\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175320    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee2e057-5724-4f65-bfe4-46cf91386696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee2e057-5724-4f65-bfe4-46cf91386696-istiod-ca-cert\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175355    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-envoy\") pod \"7ee2e057-5724-4f65-bfe4-46cf91386696\" (UID: \"7ee2e057-5724-4f65-bfe4-46cf91386696\") "
Oct 30 17:00:52.175824 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.175636    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-data" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:52.176074 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.176049    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.176391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.176363    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-certs" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:52.176700 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.176674    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-credential-socket" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:52.177112 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.176932    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-socket" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:52.177373 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.177347    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee2e057-5724-4f65-bfe4-46cf91386696-istiod-ca-cert" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:00:52.178333 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.178308    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-podinfo" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:00:52.178612 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.178580    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-kube-api-access-9cwfq" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "kube-api-access-9cwfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:52.178837 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.178814    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-envoy" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:00:52.179602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.179578    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-token" pod "7ee2e057-5724-4f65-bfe4-46cf91386696" (UID: "7ee2e057-5724-4f65-bfe4-46cf91386696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:00:52.277764 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.277732    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.277764 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.277763    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwfq\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-kube-api-access-9cwfq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.277764 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.277772    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee2e057-5724-4f65-bfe4-46cf91386696-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.278014 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.277785    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.278014 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.277796    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.278014 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.277805    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.278014 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.277812    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee2e057-5724-4f65-bfe4-46cf91386696-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.278014 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.277820    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee2e057-5724-4f65-bfe4-46cf91386696-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:00:52.800559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.800524    4497 generic.go:356] "Generic (PLEG): container finished" podID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerID="f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4" exitCode=0
Oct 30 17:00:52.801005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.800572    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"
Oct 30 17:00:52.801005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.800605    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" event={"ID":"7ee2e057-5724-4f65-bfe4-46cf91386696","Type":"ContainerDied","Data":"f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4"}
Oct 30 17:00:52.801005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.800660    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn" event={"ID":"7ee2e057-5724-4f65-bfe4-46cf91386696","Type":"ContainerDied","Data":"60c3e85e1ce15f34a0878672b05f49ea15552be8ce1c1465e2bf988160287a85"}
Oct 30 17:00:52.801005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.800691    4497 scope.go:122] "RemoveContainer" containerID="f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4"
Oct 30 17:00:52.803033 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.802985    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" event={"ID":"9905c187-904a-44ee-ab84-f6e767fc887b","Type":"ContainerStarted","Data":"5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b"}
Oct 30 17:00:52.811043 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.811024    4497 scope.go:122] "RemoveContainer" containerID="f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4"
Oct 30 17:00:52.811323 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:00:52.811303    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4\": container with ID starting with f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4 not found: ID does not exist" containerID="f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4"
Oct 30 17:00:52.811375 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.811331    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4"} err="failed to get container status \"f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4\": rpc error: code = NotFound desc = could not find container \"f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4\": container with ID starting with f206ee696340346ec59a67def9466f658360324476d490e6fcd449bdf7e679c4 not found: ID does not exist"
Oct 30 17:00:52.899346 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.899254    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"]
Oct 30 17:00:52.902642 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:52.902600    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-m6bmn"]
Oct 30 17:00:53.256238 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:53.256174    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 17:00:53.257339 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:53.257313    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:00:53.257459 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:53.257365    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:00:54.255894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:54.255848    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:00:54.256452 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:54.255928    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:00:54.302850 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:54.302816    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" path="/var/lib/kubelet/pods/7ee2e057-5724-4f65-bfe4-46cf91386696/volumes"
Oct 30 17:00:55.255832 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:55.255797    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:00:55.256010 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:55.255848    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:00:56.151737 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:56.151703    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 17:00:56.255566 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:56.255531    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:00:56.255743 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:56.255581    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:00:57.255756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:57.255722    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 17:00:57.256216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:57.255788    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 17:00:58.396704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:58.396664    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 17:00:58.398578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:00:58.398555    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 17:01:02.966901 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:02.966810    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"]
Oct 30 17:01:02.967489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:02.967473    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy"
Oct 30 17:01:02.967489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:02.967488    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy"
Oct 30 17:01:02.967643 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:02.967631    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee2e057-5724-4f65-bfe4-46cf91386696" containerName="istio-proxy"
Oct 30 17:01:02.971029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:02.971010    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:02.983379 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:02.983355    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2589b5-serving-cert\""
Oct 30 17:01:02.996423 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:02.996397    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"]
Oct 30 17:01:03.040858 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.040822    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaa1be-6f0b-451f-9486-f631143c55ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.041029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.040866    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.041029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.041007    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-metrics-certs\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.041227 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.041208    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2dr6\" (UniqueName: \"kubernetes.io/projected/b8aaa1be-6f0b-451f-9486-f631143c55ac-kube-api-access-s2dr6\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.142196 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.142148    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2dr6\" (UniqueName: \"kubernetes.io/projected/b8aaa1be-6f0b-451f-9486-f631143c55ac-kube-api-access-s2dr6\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.142388 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.142252    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaa1be-6f0b-451f-9486-f631143c55ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.142388 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.142282    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.142388 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.142372    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-metrics-certs\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.142981 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.142926    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaa1be-6f0b-451f-9486-f631143c55ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.145078 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.145053    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.145181 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.145127    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-metrics-certs\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.165021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.164994    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2dr6\" (UniqueName: \"kubernetes.io/projected/b8aaa1be-6f0b-451f-9486-f631143c55ac-kube-api-access-s2dr6\") pod \"maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.286483 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.285596    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:03.593359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.593285    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"]
Oct 30 17:01:03.596768 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:01:03.596735    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8aaa1be_6f0b_451f_9486_f631143c55ac.slice/crio-4483690b0d02d3048b8f27845365ca476a0703d971ddd2b910dc0262261b36cc WatchSource:0}: Error finding container 4483690b0d02d3048b8f27845365ca476a0703d971ddd2b910dc0262261b36cc: Status 404 returned error can't find the container with id 4483690b0d02d3048b8f27845365ca476a0703d971ddd2b910dc0262261b36cc
Oct 30 17:01:03.876529 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:03.876443    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5" event={"ID":"b8aaa1be-6f0b-451f-9486-f631143c55ac","Type":"ContainerStarted","Data":"4483690b0d02d3048b8f27845365ca476a0703d971ddd2b910dc0262261b36cc"}
Oct 30 17:01:04.524465 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.524423    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"]
Oct 30 17:01:04.527932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.527910    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:04.550102 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.550069    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"]
Oct 30 17:01:04.554451 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.554429    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-81733d\""
Oct 30 17:01:04.562508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.562467    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd292422-c1b1-4171-8860-93d71ac38c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb\" (UID: \"fd292422-c1b1-4171-8860-93d71ac38c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:04.562786 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.562684    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqdf\" (UniqueName: \"kubernetes.io/projected/fd292422-c1b1-4171-8860-93d71ac38c4e-kube-api-access-6jqdf\") pod \"payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb\" (UID: \"fd292422-c1b1-4171-8860-93d71ac38c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:04.562786 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.562723    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-81733d-dockercfg-9cgz4\""
Oct 30 17:01:04.664499 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.664464    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd292422-c1b1-4171-8860-93d71ac38c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb\" (UID: \"fd292422-c1b1-4171-8860-93d71ac38c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:04.664717 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.664605    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqdf\" (UniqueName: \"kubernetes.io/projected/fd292422-c1b1-4171-8860-93d71ac38c4e-kube-api-access-6jqdf\") pod \"payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb\" (UID: \"fd292422-c1b1-4171-8860-93d71ac38c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:04.665307 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.665288    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd292422-c1b1-4171-8860-93d71ac38c4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb\" (UID: \"fd292422-c1b1-4171-8860-93d71ac38c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:04.692337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.692308    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqdf\" (UniqueName: \"kubernetes.io/projected/fd292422-c1b1-4171-8860-93d71ac38c4e-kube-api-access-6jqdf\") pod \"payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb\" (UID: \"fd292422-c1b1-4171-8860-93d71ac38c4e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:04.843261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.842612    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:04.887780 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.886995    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5" event={"ID":"b8aaa1be-6f0b-451f-9486-f631143c55ac","Type":"ContainerStarted","Data":"1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6"}
Oct 30 17:01:04.887780 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.887293    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:04.931848 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:04.931798    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5" podStartSLOduration=2.558410272 podStartE2EDuration="2.931781434s" totalImagesPullingTime="373.371162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:04.931716713 +0000 UTC m=+2294.785594978" watchObservedRunningTime="2026-10-30 17:01:04.931781434 +0000 UTC m=+2294.785659698"
Oct 30 17:01:05.160552 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:05.160469    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"]
Oct 30 17:01:05.163990 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:01:05.163960    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd292422_c1b1_4171_8860_93d71ac38c4e.slice/crio-d6b8237920ad11dcf065c931afe38fb125335f72dd743d6da85a6856317b9508 WatchSource:0}: Error finding container d6b8237920ad11dcf065c931afe38fb125335f72dd743d6da85a6856317b9508: Status 404 returned error can't find the container with id d6b8237920ad11dcf065c931afe38fb125335f72dd743d6da85a6856317b9508
Oct 30 17:01:05.903732 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:05.903669    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb" event={"ID":"fd292422-c1b1-4171-8860-93d71ac38c4e","Type":"ContainerStarted","Data":"1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a"}
Oct 30 17:01:05.904230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:05.903786    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb" event={"ID":"fd292422-c1b1-4171-8860-93d71ac38c4e","Type":"ContainerStarted","Data":"d6b8237920ad11dcf065c931afe38fb125335f72dd743d6da85a6856317b9508"}
Oct 30 17:01:05.904230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:05.903911    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:05.939860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:05.939807    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb" podStartSLOduration=1.939794274 podStartE2EDuration="1.939794274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:05.938444938 +0000 UTC m=+2295.792323206" watchObservedRunningTime="2026-10-30 17:01:05.939794274 +0000 UTC m=+2295.793672539"
Oct 30 17:01:06.882465 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:06.882427    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"]
Oct 30 17:01:06.886594 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:06.886575    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:06.898684 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:06.898661    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-abda73-dockercfg-hw9d4\""
Oct 30 17:01:06.912845 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:06.912815    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"]
Oct 30 17:01:06.954520 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:06.954485    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-abda73\""
Oct 30 17:01:07.026225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.018536    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf\" (UID: \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:07.026225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.018753    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2p46\" (UniqueName: \"kubernetes.io/projected/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-kube-api-access-b2p46\") pod \"payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf\" (UID: \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:07.119792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.119752    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2p46\" (UniqueName: \"kubernetes.io/projected/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-kube-api-access-b2p46\") pod \"payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf\" (UID: \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:07.119973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.119956    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf\" (UID: \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:07.120727 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.120687    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf\" (UID: \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:07.140325 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.140262    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2p46\" (UniqueName: \"kubernetes.io/projected/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-kube-api-access-b2p46\") pod \"payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf\" (UID: \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:07.199179 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.199148    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:07.517083 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.517009    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"]
Oct 30 17:01:07.520091 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:01:07.520066    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda124b7a1_5f18_4779_a839_3c7a8c37c5f8.slice/crio-54abfbb84b262e8eb878fcce50bd4f70fde7ada42f06c70d734dbf563075d8df WatchSource:0}: Error finding container 54abfbb84b262e8eb878fcce50bd4f70fde7ada42f06c70d734dbf563075d8df: Status 404 returned error can't find the container with id 54abfbb84b262e8eb878fcce50bd4f70fde7ada42f06c70d734dbf563075d8df
Oct 30 17:01:07.922072 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.921970    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf" event={"ID":"a124b7a1-5f18-4779-a839-3c7a8c37c5f8","Type":"ContainerStarted","Data":"16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0"}
Oct 30 17:01:07.922072 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.922030    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf" event={"ID":"a124b7a1-5f18-4779-a839-3c7a8c37c5f8","Type":"ContainerStarted","Data":"54abfbb84b262e8eb878fcce50bd4f70fde7ada42f06c70d734dbf563075d8df"}
Oct 30 17:01:07.922072 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.922051    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:07.962477 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:07.962428    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf" podStartSLOduration=1.962413238 podStartE2EDuration="1.962413238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:07.960991103 +0000 UTC m=+2297.814869354" watchObservedRunningTime="2026-10-30 17:01:07.962413238 +0000 UTC m=+2297.816291502"
Oct 30 17:01:09.172938 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.172901    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"]
Oct 30 17:01:09.176230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.176213    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.206349 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.206316    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-3106d6-serving-cert\""
Oct 30 17:01:09.211520 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.211485    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"]
Oct 30 17:01:09.270121 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.270090    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f6f6\" (UniqueName: \"kubernetes.io/projected/2cbbe2ac-a256-465f-a5b4-f80112cd862c-kube-api-access-7f6f6\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.270312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.270146    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.270360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.270297    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbbe2ac-a256-465f-a5b4-f80112cd862c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.270424 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.270407    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.371206 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.371161    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.371375 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.371322    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbbe2ac-a256-465f-a5b4-f80112cd862c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.371464 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.371419    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.371885 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.371738    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6f6\" (UniqueName: \"kubernetes.io/projected/2cbbe2ac-a256-465f-a5b4-f80112cd862c-kube-api-access-7f6f6\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.372103 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.372058    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbbe2ac-a256-465f-a5b4-f80112cd862c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.373818 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.373797    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-metrics-certs\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.374138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.374117    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.393108 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.393082    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f6f6\" (UniqueName: \"kubernetes.io/projected/2cbbe2ac-a256-465f-a5b4-f80112cd862c-kube-api-access-7f6f6\") pod \"maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.488491 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.488388    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:09.842527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.842503    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"]
Oct 30 17:01:09.845351 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:01:09.845320    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbbe2ac_a256_465f_a5b4_f80112cd862c.slice/crio-22da5f303d31c7761055edf6eefc96bb8209aaa791396436dd330d28c2cc804d WatchSource:0}: Error finding container 22da5f303d31c7761055edf6eefc96bb8209aaa791396436dd330d28c2cc804d: Status 404 returned error can't find the container with id 22da5f303d31c7761055edf6eefc96bb8209aaa791396436dd330d28c2cc804d
Oct 30 17:01:09.936179 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:09.936140    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62" event={"ID":"2cbbe2ac-a256-465f-a5b4-f80112cd862c","Type":"ContainerStarted","Data":"22da5f303d31c7761055edf6eefc96bb8209aaa791396436dd330d28c2cc804d"}
Oct 30 17:01:10.909758 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:10.909728    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:01:10.944508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:10.944473    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62" event={"ID":"2cbbe2ac-a256-465f-a5b4-f80112cd862c","Type":"ContainerStarted","Data":"c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760"}
Oct 30 17:01:10.944681 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:10.944590    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:11.003301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:11.003238    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62" podStartSLOduration=1.663659655 podStartE2EDuration="2.003219187s" totalImagesPullingTime="339.559532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:10.989774206 +0000 UTC m=+2300.843652470" watchObservedRunningTime="2026-10-30 17:01:11.003219187 +0000 UTC m=+2300.857097459"
Oct 30 17:01:16.917167 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:16.917133    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:01:16.957428 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:16.957396    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:01:18.931783 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:18.931702    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:01:19.043426 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:19.043394    4497 generic.go:356] "Generic (PLEG): container finished" podID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerID="be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00" exitCode=0
Oct 30 17:01:19.043599 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:19.043463    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" event={"ID":"639dc55b-adf9-4a59-afc0-29f509b002b7","Type":"ContainerDied","Data":"be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00"}
Oct 30 17:01:19.043927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:19.043894    4497 scope.go:122] "RemoveContainer" containerID="be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00"
Oct 30 17:01:19.045820 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:19.045789    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:19.045898 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:19.045870    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:19.045937 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:19.045905    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:20.052805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:20.052771    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" event={"ID":"639dc55b-adf9-4a59-afc0-29f509b002b7","Type":"ContainerStarted","Data":"f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba"}
Oct 30 17:01:20.366965 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:20.366877    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:01:20.368048 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:20.368023    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 17:01:20.368211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:20.368068    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 17:01:21.324261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:21.324231    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:01:21.367605 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:21.367575    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 17:01:21.367910 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:21.367624    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 17:01:21.924031 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:01:21.923992    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:01:22.367468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:22.367426    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 17:01:22.367889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:22.367500    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 17:01:23.367247 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:23.367208    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 17:01:23.367427 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:23.367265    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 17:01:24.367244 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:24.367207    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 17:01:24.367681 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:24.367260    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 17:01:25.372152 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:25.372121    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:01:25.373337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:25.373316    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:01:40.979620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:40.979585    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"]
Oct 30 17:01:40.985215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:40.985164    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:40.997539 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:40.997515    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-abda73-serving-cert\""
Oct 30 17:01:41.022108 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.022063    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"]
Oct 30 17:01:41.087565 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.087524    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-maas-api-tls\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.087743 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.087595    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd864eb4-5923-49b8-abd6-c62b8b5e4420-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.087841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.087812    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-metrics-certs\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.087963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.087950    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvkp\" (UniqueName: \"kubernetes.io/projected/bd864eb4-5923-49b8-abd6-c62b8b5e4420-kube-api-access-5qvkp\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.188853 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.188808    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-maas-api-tls\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.189055 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.188887    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd864eb4-5923-49b8-abd6-c62b8b5e4420-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.189055 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.189028    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-metrics-certs\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.189215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.189111    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvkp\" (UniqueName: \"kubernetes.io/projected/bd864eb4-5923-49b8-abd6-c62b8b5e4420-kube-api-access-5qvkp\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.190085 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.190027    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd864eb4-5923-49b8-abd6-c62b8b5e4420-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.192228 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.192170    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-maas-api-tls\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.192689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.192665    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-metrics-certs\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.216483 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.213351    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvkp\" (UniqueName: \"kubernetes.io/projected/bd864eb4-5923-49b8-abd6-c62b8b5e4420-kube-api-access-5qvkp\") pod \"maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.302408 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.302368    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:41.613771 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:41.613681    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"]
Oct 30 17:01:42.321076 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:42.320937    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7" event={"ID":"bd864eb4-5923-49b8-abd6-c62b8b5e4420","Type":"ContainerStarted","Data":"115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58"}
Oct 30 17:01:42.321076 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:42.321023    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7" event={"ID":"bd864eb4-5923-49b8-abd6-c62b8b5e4420","Type":"ContainerStarted","Data":"89f6e3e0951627884116ac83a0a073277e0ab7a6a7570b06fbfe4867445da218"}
Oct 30 17:01:42.361480 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:42.361406    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7" podStartSLOduration=2.057845266 podStartE2EDuration="2.36138703s" totalImagesPullingTime="303.541764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:42.360387206 +0000 UTC m=+2332.214265470" watchObservedRunningTime="2026-10-30 17:01:42.36138703 +0000 UTC m=+2332.215265295"
Oct 30 17:01:43.329898 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.329865    4497 generic.go:356] "Generic (PLEG): container finished" podID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerID="19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15" exitCode=0
Oct 30 17:01:43.330400 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.329937    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" event={"ID":"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0","Type":"ContainerDied","Data":"19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15"}
Oct 30 17:01:43.330496 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.330456    4497 scope.go:122] "RemoveContainer" containerID="19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15"
Oct 30 17:01:43.331732 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.331710    4497 generic.go:356] "Generic (PLEG): container finished" podID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerID="ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970" exitCode=0
Oct 30 17:01:43.331822 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.331787    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" event={"ID":"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0","Type":"ContainerDied","Data":"ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970"}
Oct 30 17:01:43.331999 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.331985    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:43.332283 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.332257    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:43.332337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.332324    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:43.332389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.332358    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:43.332389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.332364    4497 scope.go:122] "RemoveContainer" containerID="ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970"
Oct 30 17:01:43.333853 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.333814    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:43.333975 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.333904    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:43.333975 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:43.333956    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:01:44.340731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:44.340694    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" event={"ID":"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0","Type":"ContainerStarted","Data":"9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999"}
Oct 30 17:01:44.343864 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:44.343834    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" event={"ID":"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0","Type":"ContainerStarted","Data":"66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800"}
Oct 30 17:01:44.955197 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:44.955163    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:01:44.955404 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:44.955335    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 17:01:44.955457 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:44.955397    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 17:01:45.337900 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:45.336349    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:01:45.337900 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:45.337437    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 17:01:45.337900 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:45.337504    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 17:01:45.554211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:45.546397    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:01:45.955773 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:45.955734    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 17:01:45.955969 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:45.955795    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 17:01:46.334869 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:46.334831    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 17:01:46.335044 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:46.334899    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 17:01:46.955605 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:46.955575    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 17:01:46.956068 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:46.956044    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 17:01:47.335033 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:47.334999    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 17:01:47.335293 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:47.335048    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 17:01:47.955413 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:47.955381    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 17:01:47.955635 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:47.955604    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 17:01:48.335431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:48.335399    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 17:01:48.335651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:48.335455    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 17:01:48.954864 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:48.954812    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 17:01:48.955111 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:48.954876    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 17:01:49.335442 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:49.335411    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 17:01:49.335842 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:49.335461    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 17:01:49.349687 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:49.349659    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:01:50.063391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:50.063358    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:01:50.063866 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:50.063841    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:01:50.073569 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:50.073533    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:01:50.340977 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:50.340896    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:01:50.341906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:50.341885    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:01:51.916514 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:01:51.916474    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:01:58.129108 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.129075    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"]
Oct 30 17:01:58.133543 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.133526    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.156776 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.156749    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-13c495-serving-cert\""
Oct 30 17:01:58.161221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.161177    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"]
Oct 30 17:01:58.312528 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.312489    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.312731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.312542    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.312838 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.312816    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dttpq\" (UniqueName: \"kubernetes.io/projected/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-kube-api-access-dttpq\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.312928 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.312912    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-metrics-certs\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.414885 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.414793    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dttpq\" (UniqueName: \"kubernetes.io/projected/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-kube-api-access-dttpq\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.414885 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.414865    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-metrics-certs\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.415156 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.414924    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.415156 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.414973    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.415718 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.415693    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.417501 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.417477    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-metrics-certs\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.417649 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.417628    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.436459 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.436418    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dttpq\" (UniqueName: \"kubernetes.io/projected/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-kube-api-access-dttpq\") pod \"maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.445944 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.445915    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:58.751917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:58.751825    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"]
Oct 30 17:01:58.755158 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:01:58.755128    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c661b8_5a2a_4bb9_b12a_deb30e055e0b.slice/crio-75643d0728b88106bff39e90af649a0c763061dd76c3fb4c854dfc016802eee5 WatchSource:0}: Error finding container 75643d0728b88106bff39e90af649a0c763061dd76c3fb4c854dfc016802eee5: Status 404 returned error can't find the container with id 75643d0728b88106bff39e90af649a0c763061dd76c3fb4c854dfc016802eee5
Oct 30 17:01:59.502512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:59.502428    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9" event={"ID":"48c661b8-5a2a-4bb9-b12a-deb30e055e0b","Type":"ContainerStarted","Data":"2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9"}
Oct 30 17:01:59.502512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:59.502499    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:01:59.502913 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:59.502524    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9" event={"ID":"48c661b8-5a2a-4bb9-b12a-deb30e055e0b","Type":"ContainerStarted","Data":"75643d0728b88106bff39e90af649a0c763061dd76c3fb4c854dfc016802eee5"}
Oct 30 17:01:59.541163 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:01:59.541111    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9" podStartSLOduration=1.166014178 podStartE2EDuration="1.54109724s" totalImagesPullingTime="375.083062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:59.539623108 +0000 UTC m=+2349.393501373" watchObservedRunningTime="2026-10-30 17:01:59.54109724 +0000 UTC m=+2349.394975505"
Oct 30 17:02:05.515723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:05.515691    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:02:13.707825 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.707786    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"]
Oct 30 17:02:13.712527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.712502    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.742478 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.742444    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"]
Oct 30 17:02:13.812156 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.812124    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-81733d-openshift-default-dockercfg-6hh6m\""
Oct 30 17:02:13.818872 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.818832    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfmj\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-kube-api-access-znfmj\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.819016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.818966    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.819016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.819006    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.819124 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.819044    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.819159 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.819134    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.819217 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.819205    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.819309 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.819286    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.819355 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.819331    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1365c19f-effa-4593-9397-2b8f45b7bec6-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.819414 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.819397    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920332    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920394    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920445    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920503    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920558    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920585    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1365c19f-effa-4593-9397-2b8f45b7bec6-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920629    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920825 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920725    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfmj\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-kube-api-access-znfmj\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920825 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920790    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920825 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920806    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920991 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920963    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.920991 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.920981    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.921136 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.921107    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.921382 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.921364    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1365c19f-effa-4593-9397-2b8f45b7bec6-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.922968 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.922944    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.923080 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.922949    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.941238 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.941209    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:13.941570 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:13.941546    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfmj\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-kube-api-access-znfmj\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ll5qq\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:14.029932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:14.029900    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:14.351137 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:14.351071    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"]
Oct 30 17:02:14.354136 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:02:14.354102    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1365c19f_effa_4593_9397_2b8f45b7bec6.slice/crio-97f4af66cba9237de880d93a7208a483b99f34d169d6dac6af2a4ff1734b2fde WatchSource:0}: Error finding container 97f4af66cba9237de880d93a7208a483b99f34d169d6dac6af2a4ff1734b2fde: Status 404 returned error can't find the container with id 97f4af66cba9237de880d93a7208a483b99f34d169d6dac6af2a4ff1734b2fde
Oct 30 17:02:14.357127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:14.357093    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:02:14.357242 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:14.357161    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:02:14.357242 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:14.357207    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:02:14.702594 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:14.702508    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" event={"ID":"1365c19f-effa-4593-9397-2b8f45b7bec6","Type":"ContainerStarted","Data":"93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f"}
Oct 30 17:02:14.702594 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:14.702581    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" event={"ID":"1365c19f-effa-4593-9397-2b8f45b7bec6","Type":"ContainerStarted","Data":"97f4af66cba9237de880d93a7208a483b99f34d169d6dac6af2a4ff1734b2fde"}
Oct 30 17:02:14.745787 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:14.745730    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podStartSLOduration=1.7457133 podStartE2EDuration="1.7457133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:14.742844465 +0000 UTC m=+2364.596722729" watchObservedRunningTime="2026-10-30 17:02:14.7457133 +0000 UTC m=+2364.599591563"
Oct 30 17:02:15.030534 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:15.030495    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:15.032571 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:15.032542    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 17:02:15.032723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:15.032590    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 17:02:16.030423 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:16.030385    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 17:02:16.030849 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:16.030444    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 17:02:17.030781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:17.030746    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 17:02:17.031241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:17.030806    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 17:02:18.030517 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:18.030476    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 17:02:18.030711 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:18.030533    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 17:02:19.030663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:19.030621    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 17:02:19.031116 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:19.030684    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 17:02:20.030942 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:20.030903    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 17:02:20.031383 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:20.030957    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 17:02:21.035889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:21.035857    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:21.036356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:21.036209    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:21.036964 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:21.036948    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:02:21.917300 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:02:21.917269    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:02:28.386233 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.386166    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"]
Oct 30 17:02:28.389629 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.389609    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.404715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.404688    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-5c6b3c-serving-cert\""
Oct 30 17:02:28.420960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.420919    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"]
Oct 30 17:02:28.480004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.479965    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.480004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.480006    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gwd\" (UniqueName: \"kubernetes.io/projected/f342d24d-33d2-4428-83e8-4d5d8eeee05d-kube-api-access-76gwd\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.480251 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.480236    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-metrics-certs\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.480295 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.480264    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f342d24d-33d2-4428-83e8-4d5d8eeee05d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.581956 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.581914    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.581956 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.581956    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gwd\" (UniqueName: \"kubernetes.io/projected/f342d24d-33d2-4428-83e8-4d5d8eeee05d-kube-api-access-76gwd\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.582230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.582126    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-metrics-certs\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.582230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.582152    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f342d24d-33d2-4428-83e8-4d5d8eeee05d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.582849 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.582822    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f342d24d-33d2-4428-83e8-4d5d8eeee05d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.584575 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.584553    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.584707 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.584686    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-metrics-certs\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.611924 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.611855    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gwd\" (UniqueName: \"kubernetes.io/projected/f342d24d-33d2-4428-83e8-4d5d8eeee05d-kube-api-access-76gwd\") pod \"maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:28.704641 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:28.704547    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:29.019741 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:29.019715    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"]
Oct 30 17:02:29.022726 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:02:29.022693    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf342d24d_33d2_4428_83e8_4d5d8eeee05d.slice/crio-c5f5458f1cbdcf8e494252e9a64814e91faafd509dd2da44066ff50ae99e7c2f WatchSource:0}: Error finding container c5f5458f1cbdcf8e494252e9a64814e91faafd509dd2da44066ff50ae99e7c2f: Status 404 returned error can't find the container with id c5f5458f1cbdcf8e494252e9a64814e91faafd509dd2da44066ff50ae99e7c2f
Oct 30 17:02:29.818317 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:29.818282    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x" event={"ID":"f342d24d-33d2-4428-83e8-4d5d8eeee05d","Type":"ContainerStarted","Data":"159298503eace3d4c195973489987e3184877ae728daa7fcff04b9dbdae9d661"}
Oct 30 17:02:29.818751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:29.818341    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x" event={"ID":"f342d24d-33d2-4428-83e8-4d5d8eeee05d","Type":"ContainerStarted","Data":"c5f5458f1cbdcf8e494252e9a64814e91faafd509dd2da44066ff50ae99e7c2f"}
Oct 30 17:02:29.818751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:29.818390    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:29.859853 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:29.859799    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x" podStartSLOduration=1.558582278 podStartE2EDuration="1.85978533s" totalImagesPullingTime="301.203052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:29.859618749 +0000 UTC m=+2379.713497013" watchObservedRunningTime="2026-10-30 17:02:29.85978533 +0000 UTC m=+2379.713663616"
Oct 30 17:02:30.632069 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.632036    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"]
Oct 30 17:02:30.635425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.635403    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.647692 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.647668    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"routing-overlay\""
Oct 30 17:02:30.647829 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.647790    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"kube-root-ca.crt\""
Oct 30 17:02:30.647900 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.647839    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"payload-processing-external-model-plugins-e2e-extmod-b-abda73\""
Oct 30 17:02:30.650381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.650366    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"openshift-service-ca.crt\""
Oct 30 17:02:30.657657 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.657635    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"]
Oct 30 17:02:30.744216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.742456    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"payload-processing-external-model-e2e--dc82f608-dockercfg-sdnt8\""
Oct 30 17:02:30.744216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.742556    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"agc-cred-61b8dd482a81\""
Oct 30 17:02:30.744216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.743620    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.744216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.743669    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.744216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.743711    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znkwz\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-kube-api-access-znkwz\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.744216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.743861    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.846352 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.846315    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.847016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.846989    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.847225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.847172    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znkwz\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-kube-api-access-znkwz\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.847626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.847598    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.848223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.848178    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.848367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.848341    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.849711 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.849690    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.869398 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.869361    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znkwz\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-kube-api-access-znkwz\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:30.950699 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:30.950617    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:31.258567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:31.258536    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"]
Oct 30 17:02:31.262152 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:02:31.262126    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4979ad7d_53f6_4a4c_99ca_a5f3f74e0136.slice/crio-6cd345815449e7c10b8ac8326fd2e0caf40eb832ed16dd420a935ab36ac4dad4 WatchSource:0}: Error finding container 6cd345815449e7c10b8ac8326fd2e0caf40eb832ed16dd420a935ab36ac4dad4: Status 404 returned error can't find the container with id 6cd345815449e7c10b8ac8326fd2e0caf40eb832ed16dd420a935ab36ac4dad4
Oct 30 17:02:31.835097 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:31.835066    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l" event={"ID":"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136","Type":"ContainerStarted","Data":"877015ba6215dbf3b4d42b0aaab4495687219bfc7c9e9d8185c59a4b8c09a103"}
Oct 30 17:02:31.835326 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:31.835115    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l" event={"ID":"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136","Type":"ContainerStarted","Data":"6cd345815449e7c10b8ac8326fd2e0caf40eb832ed16dd420a935ab36ac4dad4"}
Oct 30 17:02:31.835387 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:31.835358    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:31.876940 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:31.876861    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l" podStartSLOduration=1.8768428849999998 podStartE2EDuration="1.876842885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:31.874770358 +0000 UTC m=+2381.728648623" watchObservedRunningTime="2026-10-30 17:02:31.876842885 +0000 UTC m=+2381.730721151"
Oct 30 17:02:35.833149 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:35.833120    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:02:42.845482 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:42.845447    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:02:43.834077 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.834039    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"]
Oct 30 17:02:43.839699 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.839677    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.866440 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.866405    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"]
Oct 30 17:02:43.937691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.937657    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.937903 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.937816    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.937903 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.937865    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.938030 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.938014    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.938141 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.938120    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d90a75-5c79-4333-90e6-83f92002b9ef-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.938222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.938164    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.938303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.938281    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.938369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.938357    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-kube-api-access-zg2wm\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:43.938417 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:43.938382    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040264 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040234    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040300    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d90a75-5c79-4333-90e6-83f92002b9ef-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040327    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040356    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040433    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-kube-api-access-zg2wm\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040458    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040740 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040538    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040740 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040627    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040740 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040650    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040691    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.040908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.040783    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.041074 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.041046    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.041074 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.041066    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d90a75-5c79-4333-90e6-83f92002b9ef-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.041254 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.041118    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.043372 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.043346    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.043458 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.043439    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.063973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.063948    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-kube-api-access-zg2wm\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.064490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.064468    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rb4hv\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.156135 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.156043    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:44.487124 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.487100    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"]
Oct 30 17:02:44.491158 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:02:44.491125    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d90a75_5c79_4333_90e6_83f92002b9ef.slice/crio-1bf96c0cd2da9b6ed058063860a8cc6914e72b1ae50a6b6cd13bfd80dab560d3 WatchSource:0}: Error finding container 1bf96c0cd2da9b6ed058063860a8cc6914e72b1ae50a6b6cd13bfd80dab560d3: Status 404 returned error can't find the container with id 1bf96c0cd2da9b6ed058063860a8cc6914e72b1ae50a6b6cd13bfd80dab560d3
Oct 30 17:02:44.496175 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.496136    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:02:44.496315 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.496233    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:02:44.496315 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.496267    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:02:44.940932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.940898    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" event={"ID":"06d90a75-5c79-4333-90e6-83f92002b9ef","Type":"ContainerStarted","Data":"c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203"}
Oct 30 17:02:44.941464 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:44.940946    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" event={"ID":"06d90a75-5c79-4333-90e6-83f92002b9ef","Type":"ContainerStarted","Data":"1bf96c0cd2da9b6ed058063860a8cc6914e72b1ae50a6b6cd13bfd80dab560d3"}
Oct 30 17:02:45.000090 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:45.000019    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podStartSLOduration=1.999996392 podStartE2EDuration="1.999996392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:44.995317252 +0000 UTC m=+2394.849195516" watchObservedRunningTime="2026-10-30 17:02:44.999996392 +0000 UTC m=+2394.853874721"
Oct 30 17:02:45.156239 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:45.156200    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:45.157651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:45.157616    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 17:02:45.157770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:45.157666    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 17:02:46.157229 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:46.157173    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 17:02:46.157721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:46.157259    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 17:02:47.171231 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:47.166762    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 17:02:47.171231 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:47.166828    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 17:02:48.157337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:48.157301    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 17:02:48.157622 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:48.157370    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 17:02:49.157398 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:49.157357    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 17:02:49.157828 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:49.157423    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 17:02:50.156741 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:50.156701    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 30 17:02:50.156913 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:50.156761    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 30 17:02:51.161792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:51.161758    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:51.162230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:51.162130    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:51.162897 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:02:51.162877    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:02:51.917499 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:02:51.917460    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:03:03.709865 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:03.709805    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"]
Oct 30 17:03:08.710377 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:08.710324    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb" podUID="fd292422-c1b1-4171-8860-93d71ac38c4e" containerName="payload-pre-processing" containerID="cri-o://1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a" gracePeriod=25
Oct 30 17:03:08.993468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:08.993441    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:03:09.026788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.026755    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd292422-c1b1-4171-8860-93d71ac38c4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd292422-c1b1-4171-8860-93d71ac38c4e-plugins-config-volume\") pod \"fd292422-c1b1-4171-8860-93d71ac38c4e\" (UID: \"fd292422-c1b1-4171-8860-93d71ac38c4e\") "
Oct 30 17:03:09.027019 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.027000    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd292422-c1b1-4171-8860-93d71ac38c4e-kube-api-access-6jqdf\" (UniqueName: \"kubernetes.io/projected/fd292422-c1b1-4171-8860-93d71ac38c4e-kube-api-access-6jqdf\") pod \"fd292422-c1b1-4171-8860-93d71ac38c4e\" (UID: \"fd292422-c1b1-4171-8860-93d71ac38c4e\") "
Oct 30 17:03:09.027171 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.027146    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd292422-c1b1-4171-8860-93d71ac38c4e-plugins-config-volume" pod "fd292422-c1b1-4171-8860-93d71ac38c4e" (UID: "fd292422-c1b1-4171-8860-93d71ac38c4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:03:09.027648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.027622    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd292422-c1b1-4171-8860-93d71ac38c4e-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:09.029220 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.029167    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd292422-c1b1-4171-8860-93d71ac38c4e-kube-api-access-6jqdf" pod "fd292422-c1b1-4171-8860-93d71ac38c4e" (UID: "fd292422-c1b1-4171-8860-93d71ac38c4e"). InnerVolumeSpecName "kube-api-access-6jqdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:03:09.129640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.129605    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jqdf\" (UniqueName: \"kubernetes.io/projected/fd292422-c1b1-4171-8860-93d71ac38c4e-kube-api-access-6jqdf\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:09.168364 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.168328    4497 generic.go:356] "Generic (PLEG): container finished" podID="fd292422-c1b1-4171-8860-93d71ac38c4e" containerID="1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a" exitCode=0
Oct 30 17:03:09.168531 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.168370    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"
Oct 30 17:03:09.168531 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.168415    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb" event={"ID":"fd292422-c1b1-4171-8860-93d71ac38c4e","Type":"ContainerDied","Data":"1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a"}
Oct 30 17:03:09.168531 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.168464    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb" event={"ID":"fd292422-c1b1-4171-8860-93d71ac38c4e","Type":"ContainerDied","Data":"d6b8237920ad11dcf065c931afe38fb125335f72dd743d6da85a6856317b9508"}
Oct 30 17:03:09.168531 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.168489    4497 scope.go:122] "RemoveContainer" containerID="1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a"
Oct 30 17:03:09.178959 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.178940    4497 scope.go:122] "RemoveContainer" containerID="1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a"
Oct 30 17:03:09.179293 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:03:09.179267    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a\": container with ID starting with 1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a not found: ID does not exist" containerID="1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a"
Oct 30 17:03:09.179396 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.179306    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a"} err="failed to get container status \"1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a\": rpc error: code = NotFound desc = could not find container \"1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a\": container with ID starting with 1b9b8d8863b336084a738259eb5e30c4d811c54b6d9908c047205c658987e44a not found: ID does not exist"
Oct 30 17:03:09.217081 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.216999    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"]
Oct 30 17:03:09.221055 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:09.221027    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-81733d-7f68748ff-dvxzb"]
Oct 30 17:03:10.301294 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:10.301259    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd292422-c1b1-4171-8860-93d71ac38c4e" path="/var/lib/kubelet/pods/fd292422-c1b1-4171-8860-93d71ac38c4e/volumes"
Oct 30 17:03:21.917791 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:03:21.917754    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:03:50.817852 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:50.817805    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"]
Oct 30 17:03:50.818494 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:50.818096    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy" containerID="cri-o://93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f" gracePeriod=30
Oct 30 17:03:50.833054 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:50.833021    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"]
Oct 30 17:03:50.833351 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:50.833250    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy" containerID="cri-o://c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203" gracePeriod=30
Oct 30 17:03:51.039899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:51.039867    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:03:51.040073 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:51.039934    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:03:51.165664 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:51.165581    4497 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:03:51.165664 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:51.165638    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:03:51.917956 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:03:51.917915    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:03:55.548824 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:55.548785    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"]
Oct 30 17:03:56.146060 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.145938    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:03:56.183676 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.183645    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:03:56.332476 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332440    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1365c19f-effa-4593-9397-2b8f45b7bec6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1365c19f-effa-4593-9397-2b8f45b7bec6-istiod-ca-cert\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.332663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332507    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-podinfo\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.332663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332579    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-credential-socket\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.332663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332620    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-podinfo\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.332663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332653    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-certs\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.332887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332690    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-token\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-token\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.332887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332720    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d90a75-5c79-4333-90e6-83f92002b9ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d90a75-5c79-4333-90e6-83f92002b9ef-istiod-ca-cert\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.332887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332754    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-token\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-token\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.332887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332819    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-envoy\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.333105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332894    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-data\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.333105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332902    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-credential-socket" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.333105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332933    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-socket\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.333105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.332967    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-certs\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.333105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.333031    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-credential-socket\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.333386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.333111    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-kube-api-access-zg2wm\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.333386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.333159    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-envoy\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.333386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.333263    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1365c19f-effa-4593-9397-2b8f45b7bec6-istiod-ca-cert" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:03:56.333386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.333278    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-socket\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.333386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.333293    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d90a75-5c79-4333-90e6-83f92002b9ef-istiod-ca-cert" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:03:56.333386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.333312    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-kube-api-access-znfmj\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-kube-api-access-znfmj\") pod \"1365c19f-effa-4593-9397-2b8f45b7bec6\" (UID: \"1365c19f-effa-4593-9397-2b8f45b7bec6\") "
Oct 30 17:03:56.333954 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.333781    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-certs" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.334354 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.334155    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-certs" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.334670 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.334496    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-data" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.334670 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.334501    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-data\") pod \"06d90a75-5c79-4333-90e6-83f92002b9ef\" (UID: \"06d90a75-5c79-4333-90e6-83f92002b9ef\") "
Oct 30 17:03:56.334814 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.334704    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-data" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.334880 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.334867    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-socket" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.335119 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.334961    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-credential-socket" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.335119 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335088    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-socket" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.335583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335561    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.335691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335586    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.335691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335600    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1365c19f-effa-4593-9397-2b8f45b7bec6-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.335691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335611    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.335691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335638    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.335691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335652    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06d90a75-5c79-4333-90e6-83f92002b9ef-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.335691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335664    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.335691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335675    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.335691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335687    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.336127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.335701    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.337491 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.337469    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-podinfo" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:03:56.337594 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.337524    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-token" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:03:56.337851 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.337748    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-kube-api-access-znfmj" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "kube-api-access-znfmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:03:56.337851 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.337823    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-token" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:03:56.338451 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.338425    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-podinfo" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:03:56.338889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.338861    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-envoy" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.338985 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.338888    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-kube-api-access-zg2wm" pod "06d90a75-5c79-4333-90e6-83f92002b9ef" (UID: "06d90a75-5c79-4333-90e6-83f92002b9ef"). InnerVolumeSpecName "kube-api-access-zg2wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:03:56.339128 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.339105    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-envoy" pod "1365c19f-effa-4593-9397-2b8f45b7bec6" (UID: "1365c19f-effa-4593-9397-2b8f45b7bec6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:03:56.437245 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.437135    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.437245 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.437173    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.437245 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.437228    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg2wm\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-kube-api-access-zg2wm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.437508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.437249    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.437508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.437264    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znfmj\" (UniqueName: \"kubernetes.io/projected/1365c19f-effa-4593-9397-2b8f45b7bec6-kube-api-access-znfmj\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.437508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.437280    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1365c19f-effa-4593-9397-2b8f45b7bec6-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.437508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.437310    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.437508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.437334    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06d90a75-5c79-4333-90e6-83f92002b9ef-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:03:56.669992 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.669958    4497 generic.go:356] "Generic (PLEG): container finished" podID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerID="93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f" exitCode=0
Oct 30 17:03:56.670474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.670034    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"
Oct 30 17:03:56.670474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.670041    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" event={"ID":"1365c19f-effa-4593-9397-2b8f45b7bec6","Type":"ContainerDied","Data":"93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f"}
Oct 30 17:03:56.670474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.670094    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq" event={"ID":"1365c19f-effa-4593-9397-2b8f45b7bec6","Type":"ContainerDied","Data":"97f4af66cba9237de880d93a7208a483b99f34d169d6dac6af2a4ff1734b2fde"}
Oct 30 17:03:56.670474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.670120    4497 scope.go:122] "RemoveContainer" containerID="93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f"
Oct 30 17:03:56.671862 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.671838    4497 generic.go:356] "Generic (PLEG): container finished" podID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerID="c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203" exitCode=0
Oct 30 17:03:56.672032 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.671906    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" event={"ID":"06d90a75-5c79-4333-90e6-83f92002b9ef","Type":"ContainerDied","Data":"c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203"}
Oct 30 17:03:56.672032 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.671947    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv" event={"ID":"06d90a75-5c79-4333-90e6-83f92002b9ef","Type":"ContainerDied","Data":"1bf96c0cd2da9b6ed058063860a8cc6914e72b1ae50a6b6cd13bfd80dab560d3"}
Oct 30 17:03:56.672318 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.672299    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"
Oct 30 17:03:56.683296 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.683275    4497 scope.go:122] "RemoveContainer" containerID="93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f"
Oct 30 17:03:56.683764 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:03:56.683740    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f\": container with ID starting with 93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f not found: ID does not exist" containerID="93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f"
Oct 30 17:03:56.683850 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.683774    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f"} err="failed to get container status \"93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f\": rpc error: code = NotFound desc = could not find container \"93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f\": container with ID starting with 93101bdde9e390cda52ccf455ea99499b3fc74c68fe7f759f072ca82bb9f9d5f not found: ID does not exist"
Oct 30 17:03:56.683850 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.683794    4497 scope.go:122] "RemoveContainer" containerID="c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203"
Oct 30 17:03:56.694540 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.694515    4497 scope.go:122] "RemoveContainer" containerID="c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203"
Oct 30 17:03:56.694902 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:03:56.694874    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203\": container with ID starting with c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203 not found: ID does not exist" containerID="c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203"
Oct 30 17:03:56.694972 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.694914    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203"} err="failed to get container status \"c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203\": rpc error: code = NotFound desc = could not find container \"c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203\": container with ID starting with c3c17df7abe77dcb4a9940a31aa8253b295516ef89fc3f305bc2b3e189300203 not found: ID does not exist"
Oct 30 17:03:56.718370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.718341    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"]
Oct 30 17:03:56.727712 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.727684    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rb4hv"]
Oct 30 17:03:56.751456 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.751426    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"]
Oct 30 17:03:56.763858 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:56.763833    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ll5qq"]
Oct 30 17:03:58.303518 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:58.303480    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" path="/var/lib/kubelet/pods/06d90a75-5c79-4333-90e6-83f92002b9ef/volumes"
Oct 30 17:03:58.304078 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:03:58.304062    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" path="/var/lib/kubelet/pods/1365c19f-effa-4593-9397-2b8f45b7bec6/volumes"
Oct 30 17:04:00.549428 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.549381    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l" podUID="4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" containerName="payload-processing" containerID="cri-o://877015ba6215dbf3b4d42b0aaab4495687219bfc7c9e9d8185c59a4b8c09a103" gracePeriod=25
Oct 30 17:04:00.702670 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.702636    4497 generic.go:356] "Generic (PLEG): container finished" podID="4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" containerID="877015ba6215dbf3b4d42b0aaab4495687219bfc7c9e9d8185c59a4b8c09a103" exitCode=0
Oct 30 17:04:00.702840 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.702712    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l" event={"ID":"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136","Type":"ContainerDied","Data":"877015ba6215dbf3b4d42b0aaab4495687219bfc7c9e9d8185c59a4b8c09a103"}
Oct 30 17:04:00.822419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.822343    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:04:00.911909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.911876    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-kube-api-access-znkwz\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-kube-api-access-znkwz\") pod \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") "
Oct 30 17:04:00.912096 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.912072    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-routing-overlay-volume\") pod \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") "
Oct 30 17:04:00.912151 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.912140    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-plugins-config-volume\") pod \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") "
Oct 30 17:04:00.912204 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.912164    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-provider-credentials\") pod \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\" (UID: \"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136\") "
Oct 30 17:04:00.912544 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.912511    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-routing-overlay-volume" pod "4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" (UID: "4979ad7d-53f6-4a4c-99ca-a5f3f74e0136"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:00.912544 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.912538    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-plugins-config-volume" pod "4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" (UID: "4979ad7d-53f6-4a4c-99ca-a5f3f74e0136"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:04:00.912720 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.912660    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:04:00.912720 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.912680    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:04:00.914296 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.914272    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-kube-api-access-znkwz" pod "4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" (UID: "4979ad7d-53f6-4a4c-99ca-a5f3f74e0136"). InnerVolumeSpecName "kube-api-access-znkwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:00.914386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:00.914305    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-provider-credentials" pod "4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" (UID: "4979ad7d-53f6-4a4c-99ca-a5f3f74e0136"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:04:01.014149 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:01.014099    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:04:01.014149 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:01.014131    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znkwz\" (UniqueName: \"kubernetes.io/projected/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136-kube-api-access-znkwz\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:04:01.711330 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:01.711302    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"
Oct 30 17:04:01.711769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:01.711333    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l" event={"ID":"4979ad7d-53f6-4a4c-99ca-a5f3f74e0136","Type":"ContainerDied","Data":"6cd345815449e7c10b8ac8326fd2e0caf40eb832ed16dd420a935ab36ac4dad4"}
Oct 30 17:04:01.711769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:01.711391    4497 scope.go:122] "RemoveContainer" containerID="877015ba6215dbf3b4d42b0aaab4495687219bfc7c9e9d8185c59a4b8c09a103"
Oct 30 17:04:01.756009 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:01.755944    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"]
Oct 30 17:04:01.765381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:01.765349    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49w8j9l"]
Oct 30 17:04:02.300893 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:02.300812    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" path="/var/lib/kubelet/pods/4979ad7d-53f6-4a4c-99ca-a5f3f74e0136/volumes"
Oct 30 17:04:13.338048 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338016    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"]
Oct 30 17:04:13.338676 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338661    4497 cpu_manager.go:414] "containerMap: removing container" podUID="fd292422-c1b1-4171-8860-93d71ac38c4e" containerName="payload-pre-processing"
Oct 30 17:04:13.338736 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338677    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd292422-c1b1-4171-8860-93d71ac38c4e" containerName="payload-pre-processing"
Oct 30 17:04:13.338736 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338691    4497 cpu_manager.go:414] "containerMap: removing container" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy"
Oct 30 17:04:13.338736 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338696    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy"
Oct 30 17:04:13.338736 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338727    4497 cpu_manager.go:414] "containerMap: removing container" podUID="4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" containerName="payload-processing"
Oct 30 17:04:13.338736 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338732    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" containerName="payload-processing"
Oct 30 17:04:13.338899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338752    4497 cpu_manager.go:414] "containerMap: removing container" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy"
Oct 30 17:04:13.338899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338757    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy"
Oct 30 17:04:13.338899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338858    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="4979ad7d-53f6-4a4c-99ca-a5f3f74e0136" containerName="payload-processing"
Oct 30 17:04:13.338899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338870    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d90a75-5c79-4333-90e6-83f92002b9ef" containerName="istio-proxy"
Oct 30 17:04:13.338899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338880    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd292422-c1b1-4171-8860-93d71ac38c4e" containerName="payload-pre-processing"
Oct 30 17:04:13.338899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.338887    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="1365c19f-effa-4593-9397-2b8f45b7bec6" containerName="istio-proxy"
Oct 30 17:04:13.343849 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.343828    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.356501 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.356465    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"agc-cred-61b8dd482a81\""
Oct 30 17:04:13.366125 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.366095    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"agc-cred-50cf238a3943\""
Oct 30 17:04:13.366278 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.366158    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"payload-processing-external-model-plugins-e2e-extmod-b-abda73\""
Oct 30 17:04:13.366278 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.366178    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"payload-processing-external-model-e2e--dc82f608-dockercfg-sdnt8\""
Oct 30 17:04:13.366278 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.366178    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"agc-cred-7f008a3cd8f8\""
Oct 30 17:04:13.366278 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.366248    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"routing-overlay\""
Oct 30 17:04:13.366278 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.366265    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"kube-root-ca.crt\""
Oct 30 17:04:13.366278 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.366271    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"openshift-service-ca.crt\""
Oct 30 17:04:13.371292 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.371272    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"]
Oct 30 17:04:13.416302 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.416265    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwcc\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-kube-api-access-dzwcc\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.416477 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.416382    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.416538 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.416483    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.416608 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.416573    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.517495 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.517455    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.517690 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.517505    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.517690 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.517625    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwcc\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-kube-api-access-dzwcc\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.517690 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.517679    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.518220 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.518180    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.518336 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.518269    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.520259 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.520240    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.538177 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.538151    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwcc\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-kube-api-access-dzwcc\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.658833 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.658750    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:13.967501 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:13.967419    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"]
Oct 30 17:04:13.971110 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:04:13.971084    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8be386_88b3_4154_a086_b20c722b7c3b.slice/crio-feca49a5d212192767c9db3a9298dea015ed92899d02e54075510d7ba8cf4422 WatchSource:0}: Error finding container feca49a5d212192767c9db3a9298dea015ed92899d02e54075510d7ba8cf4422: Status 404 returned error can't find the container with id feca49a5d212192767c9db3a9298dea015ed92899d02e54075510d7ba8cf4422
Oct 30 17:04:14.816471 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:14.816435    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f" event={"ID":"5f8be386-88b3-4154-a086-b20c722b7c3b","Type":"ContainerStarted","Data":"3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d"}
Oct 30 17:04:14.817020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:14.816494    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f" event={"ID":"5f8be386-88b3-4154-a086-b20c722b7c3b","Type":"ContainerStarted","Data":"feca49a5d212192767c9db3a9298dea015ed92899d02e54075510d7ba8cf4422"}
Oct 30 17:04:14.817020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:14.816535    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:14.856323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:14.856272    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f" podStartSLOduration=1.856257515 podStartE2EDuration="1.856257515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:14.851734821 +0000 UTC m=+2484.705613086" watchObservedRunningTime="2026-10-30 17:04:14.856257515 +0000 UTC m=+2484.710135780"
Oct 30 17:04:21.916744 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:04:21.916706    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:04:25.826953 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:25.826924    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:04:41.053562 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.053524    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"]
Oct 30 17:04:41.058305 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.058278    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.070851 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.070824    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-7c68f4c17b3a\""
Oct 30 17:04:41.070851 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.070827    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-3e1593902499\""
Oct 30 17:04:41.079904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.079864    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5c6b3c\""
Oct 30 17:04:41.079904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.079864    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"openshift-service-ca.crt\""
Oct 30 17:04:41.080158 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.079880    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"routing-overlay\""
Oct 30 17:04:41.080158 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.079953    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"kube-root-ca.crt\""
Oct 30 17:04:41.080158 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.080130    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-e2e--22b3ecb4-dockercfg-6tsff\""
Oct 30 17:04:41.085449 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.085413    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"]
Oct 30 17:04:41.108330 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.108295    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.108502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.108339    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.108545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.108522    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.108688 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.108672    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8vl\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-kube-api-access-cm8vl\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.218207 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.218157    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.218401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.218240    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.218401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.218340    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.218529 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.218495    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8vl\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-kube-api-access-cm8vl\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.219206 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.219152    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.219479 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.219458    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.221436 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.221409    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.239721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.239689    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8vl\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-kube-api-access-cm8vl\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.375107 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.375015    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:41.682595 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:41.682425    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"]
Oct 30 17:04:41.685893 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:04:41.685861    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e0d7ac_47a4_403b_b997_23364cfe27b4.slice/crio-18bf0f2a084091882ca42193203680a97836eb5aa3b1bd7b41825d4fbacb0701 WatchSource:0}: Error finding container 18bf0f2a084091882ca42193203680a97836eb5aa3b1bd7b41825d4fbacb0701: Status 404 returned error can't find the container with id 18bf0f2a084091882ca42193203680a97836eb5aa3b1bd7b41825d4fbacb0701
Oct 30 17:04:42.029385 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:42.029349    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5" event={"ID":"39e0d7ac-47a4-403b-b997-23364cfe27b4","Type":"ContainerStarted","Data":"c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a"}
Oct 30 17:04:42.029581 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:42.029400    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5" event={"ID":"39e0d7ac-47a4-403b-b997-23364cfe27b4","Type":"ContainerStarted","Data":"18bf0f2a084091882ca42193203680a97836eb5aa3b1bd7b41825d4fbacb0701"}
Oct 30 17:04:42.029581 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:42.029492    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:04:42.066882 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:42.066833    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5" podStartSLOduration=1.0668172519999999 podStartE2EDuration="1.066817252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:42.066069428 +0000 UTC m=+2511.919947693" watchObservedRunningTime="2026-10-30 17:04:42.066817252 +0000 UTC m=+2511.920695517"
Oct 30 17:04:51.917564 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:04:51.917521    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:04:53.039804 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:04:53.039766    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:05:04.301138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.301103    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"]
Oct 30 17:05:04.306986 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.306965    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.319685 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.319659    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"openshift-service-ca.crt\""
Oct 30 17:05:04.319826 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.319733    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2589b5\""
Oct 30 17:05:04.319826 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.319750    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"kube-root-ca.crt\""
Oct 30 17:05:04.319826 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.319779    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"routing-overlay\""
Oct 30 17:05:04.319977 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.319933    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"agc-cred-22906e10d5c1\""
Oct 30 17:05:04.323631 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.323609    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"]
Oct 30 17:05:04.329329 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.329307    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-e2e--8e3f1132-dockercfg-x82qg\""
Oct 30 17:05:04.370022 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.369989    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.370232 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.370044    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.370232 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.370108    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkvd\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-kube-api-access-wjkvd\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.370368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.370261    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.475212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.472491    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.475212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.472568    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.475212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.472617    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkvd\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-kube-api-access-wjkvd\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.475212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.472696    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.487232 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.482154    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.487232 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.482993    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.496976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.496912    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.508613 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.508579    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkvd\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-kube-api-access-wjkvd\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.621045 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.620945    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:04.936139 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:04.936116    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"]
Oct 30 17:05:04.939171 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:05:04.939147    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08da8c0d_6a8c_45ef_bb5e_d53bf799f7e5.slice/crio-f627e3e1050443fef95b80b83f3f26e519a2cee186d486f3baff9929091dd41b WatchSource:0}: Error finding container f627e3e1050443fef95b80b83f3f26e519a2cee186d486f3baff9929091dd41b: Status 404 returned error can't find the container with id f627e3e1050443fef95b80b83f3f26e519a2cee186d486f3baff9929091dd41b
Oct 30 17:05:05.200083 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:05.199979    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc" event={"ID":"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5","Type":"ContainerStarted","Data":"dc0677945e71f64c1198e7541eef551668db2e0bff574e3b9d2eeadd5c0b8147"}
Oct 30 17:05:05.200083 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:05.200049    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc" event={"ID":"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5","Type":"ContainerStarted","Data":"f627e3e1050443fef95b80b83f3f26e519a2cee186d486f3baff9929091dd41b"}
Oct 30 17:05:05.200083 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:05.200080    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:05.237470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:05.237415    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc" podStartSLOduration=1.237401322 podStartE2EDuration="1.237401322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:05.235631998 +0000 UTC m=+2535.089510263" watchObservedRunningTime="2026-10-30 17:05:05.237401322 +0000 UTC m=+2535.091279594"
Oct 30 17:05:08.827912 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:08.827876    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"]
Oct 30 17:05:08.832386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:08.832369    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:08.858887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:08.858859    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"]
Oct 30 17:05:08.942424 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:08.942387    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-kube-api-access-cgr8w\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:08.942609 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:08.942438    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:08.942655 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:08.942634    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:08.942785 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:08.942768    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.044729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.044694    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.044962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.044752    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-kube-api-access-cgr8w\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.044962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.044798    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.044962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.044933    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.045644 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.045617    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.045753 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.045618    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.047575 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.047550    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.065565 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.065538    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-kube-api-access-cgr8w\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.148163 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.148073    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:09.515316 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:09.515241    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"]
Oct 30 17:05:09.518348 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:05:09.518322    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23fc42f_30f9_4676_8394_31f2f41b8638.slice/crio-593af00af75745083f6a57f5609d5ffeaa1fb9898a1643e22ffeec8d5843063b WatchSource:0}: Error finding container 593af00af75745083f6a57f5609d5ffeaa1fb9898a1643e22ffeec8d5843063b: Status 404 returned error can't find the container with id 593af00af75745083f6a57f5609d5ffeaa1fb9898a1643e22ffeec8d5843063b
Oct 30 17:05:10.250476 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:10.250442    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn" event={"ID":"b23fc42f-30f9-4676-8394-31f2f41b8638","Type":"ContainerStarted","Data":"dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946"}
Oct 30 17:05:10.250894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:10.250499    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn" event={"ID":"b23fc42f-30f9-4676-8394-31f2f41b8638","Type":"ContainerStarted","Data":"593af00af75745083f6a57f5609d5ffeaa1fb9898a1643e22ffeec8d5843063b"}
Oct 30 17:05:10.250894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:10.250527    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:10.290297 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:10.290242    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn" podStartSLOduration=2.290226381 podStartE2EDuration="2.290226381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:10.286121557 +0000 UTC m=+2540.139999822" watchObservedRunningTime="2026-10-30 17:05:10.290226381 +0000 UTC m=+2540.144104670"
Oct 30 17:05:11.089652 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:11.089617    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"]
Oct 30 17:05:11.092615 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:11.092591    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:15.325265 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:15.325230    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"]
Oct 30 17:05:16.090671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.090611    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc" podUID="08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" containerName="payload-processing" containerID="cri-o://dc0677945e71f64c1198e7541eef551668db2e0bff574e3b9d2eeadd5c0b8147" gracePeriod=25
Oct 30 17:05:16.293284 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.293238    4497 generic.go:356] "Generic (PLEG): container finished" podID="08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" containerID="dc0677945e71f64c1198e7541eef551668db2e0bff574e3b9d2eeadd5c0b8147" exitCode=0
Oct 30 17:05:16.302496 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.302459    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc" event={"ID":"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5","Type":"ContainerDied","Data":"dc0677945e71f64c1198e7541eef551668db2e0bff574e3b9d2eeadd5c0b8147"}
Oct 30 17:05:16.370667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.370639    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:16.491274 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.491238    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-kube-api-access-wjkvd\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-kube-api-access-wjkvd\") pod \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") "
Oct 30 17:05:16.491454 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.491421    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-routing-overlay-volume\") pod \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") "
Oct 30 17:05:16.491545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.491530    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-provider-credentials\") pod \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") "
Oct 30 17:05:16.491633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.491572    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-plugins-config-volume\") pod \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\" (UID: \"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5\") "
Oct 30 17:05:16.491941 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.491910    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-routing-overlay-volume" pod "08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" (UID: "08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:16.492076 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.492028    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-plugins-config-volume" pod "08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" (UID: "08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:16.492146 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.492116    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:16.493762 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.493724    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-kube-api-access-wjkvd" pod "08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" (UID: "08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5"). InnerVolumeSpecName "kube-api-access-wjkvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:16.493892 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.493853    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-provider-credentials" pod "08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" (UID: "08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:16.593018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.592974    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:16.593018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.593012    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:16.593018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:16.593021    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjkvd\" (UniqueName: \"kubernetes.io/projected/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5-kube-api-access-wjkvd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:17.302331 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:17.302299    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"
Oct 30 17:05:17.302634 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:17.302291    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc" event={"ID":"08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5","Type":"ContainerDied","Data":"f627e3e1050443fef95b80b83f3f26e519a2cee186d486f3baff9929091dd41b"}
Oct 30 17:05:17.302634 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:17.302435    4497 scope.go:122] "RemoveContainer" containerID="dc0677945e71f64c1198e7541eef551668db2e0bff574e3b9d2eeadd5c0b8147"
Oct 30 17:05:17.349348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:17.349310    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"]
Oct 30 17:05:17.360371 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:17.360334    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7485txtsc"]
Oct 30 17:05:18.302350 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:18.302315    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" path="/var/lib/kubelet/pods/08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5/volumes"
Oct 30 17:05:20.326528 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.326485    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5" podUID="39e0d7ac-47a4-403b-b997-23364cfe27b4" containerName="payload-processing" containerID="cri-o://c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a" gracePeriod=25
Oct 30 17:05:20.601399 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.601374    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:05:20.665968 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.665931    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-plugins-config-volume\") pod \"39e0d7ac-47a4-403b-b997-23364cfe27b4\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") "
Oct 30 17:05:20.666224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.666057    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-routing-overlay-volume\") pod \"39e0d7ac-47a4-403b-b997-23364cfe27b4\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") "
Oct 30 17:05:20.666325 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.666227    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-provider-credentials\") pod \"39e0d7ac-47a4-403b-b997-23364cfe27b4\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") "
Oct 30 17:05:20.666325 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.666256    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-kube-api-access-cm8vl\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-kube-api-access-cm8vl\") pod \"39e0d7ac-47a4-403b-b997-23364cfe27b4\" (UID: \"39e0d7ac-47a4-403b-b997-23364cfe27b4\") "
Oct 30 17:05:20.666511 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.666420    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-plugins-config-volume" pod "39e0d7ac-47a4-403b-b997-23364cfe27b4" (UID: "39e0d7ac-47a4-403b-b997-23364cfe27b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:20.666818 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.666790    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-routing-overlay-volume" pod "39e0d7ac-47a4-403b-b997-23364cfe27b4" (UID: "39e0d7ac-47a4-403b-b997-23364cfe27b4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:20.666995 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.666970    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:20.667065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.667000    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39e0d7ac-47a4-403b-b997-23364cfe27b4-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:20.669260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.669224    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-kube-api-access-cm8vl" pod "39e0d7ac-47a4-403b-b997-23364cfe27b4" (UID: "39e0d7ac-47a4-403b-b997-23364cfe27b4"). InnerVolumeSpecName "kube-api-access-cm8vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:20.669561 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.669538    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-provider-credentials" pod "39e0d7ac-47a4-403b-b997-23364cfe27b4" (UID: "39e0d7ac-47a4-403b-b997-23364cfe27b4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:20.768818 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.768776    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:20.768818 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:20.768812    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8vl\" (UniqueName: \"kubernetes.io/projected/39e0d7ac-47a4-403b-b997-23364cfe27b4-kube-api-access-cm8vl\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:21.260766 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.260736    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:21.332939 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.332898    4497 generic.go:356] "Generic (PLEG): container finished" podID="39e0d7ac-47a4-403b-b997-23364cfe27b4" containerID="c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a" exitCode=0
Oct 30 17:05:21.333375 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.333002    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5" event={"ID":"39e0d7ac-47a4-403b-b997-23364cfe27b4","Type":"ContainerDied","Data":"c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a"}
Oct 30 17:05:21.333375 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.333055    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5" event={"ID":"39e0d7ac-47a4-403b-b997-23364cfe27b4","Type":"ContainerDied","Data":"18bf0f2a084091882ca42193203680a97836eb5aa3b1bd7b41825d4fbacb0701"}
Oct 30 17:05:21.333375 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.333084    4497 scope.go:122] "RemoveContainer" containerID="c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a"
Oct 30 17:05:21.333375 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.333319    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"
Oct 30 17:05:21.338421 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.338392    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"]
Oct 30 17:05:21.353104 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.353079    4497 scope.go:122] "RemoveContainer" containerID="c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a"
Oct 30 17:05:21.353538 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:05:21.353516    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a\": container with ID starting with c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a not found: ID does not exist" containerID="c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a"
Oct 30 17:05:21.353628 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.353551    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a"} err="failed to get container status \"c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a\": rpc error: code = NotFound desc = could not find container \"c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a\": container with ID starting with c7283f63acf30b485a8b2571e4c301ab027ea8fbdb46a614a7bf8c484829fc1a not found: ID does not exist"
Oct 30 17:05:21.426476 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.426438    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"]
Oct 30 17:05:21.430221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:21.430179    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd2qks5"]
Oct 30 17:05:21.917157 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:05:21.917118    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:05:22.337680 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.328612    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e0d7ac-47a4-403b-b997-23364cfe27b4" path="/var/lib/kubelet/pods/39e0d7ac-47a4-403b-b997-23364cfe27b4/volumes"
Oct 30 17:05:22.415579 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.415547    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"]
Oct 30 17:05:22.416295 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.416280    4497 cpu_manager.go:414] "containerMap: removing container" podUID="08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" containerName="payload-processing"
Oct 30 17:05:22.416342 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.416296    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" containerName="payload-processing"
Oct 30 17:05:22.416342 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.416314    4497 cpu_manager.go:414] "containerMap: removing container" podUID="39e0d7ac-47a4-403b-b997-23364cfe27b4" containerName="payload-processing"
Oct 30 17:05:22.416342 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.416321    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e0d7ac-47a4-403b-b997-23364cfe27b4" containerName="payload-processing"
Oct 30 17:05:22.416472 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.416425    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="08da8c0d-6a8c-45ef-bb5e-d53bf799f7e5" containerName="payload-processing"
Oct 30 17:05:22.416472 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.416436    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e0d7ac-47a4-403b-b997-23364cfe27b4" containerName="payload-processing"
Oct 30 17:05:22.421252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.421232    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.446555 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.446522    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"]
Oct 30 17:05:22.499672 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.499633    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.499857 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.499720    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lzn\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-kube-api-access-m2lzn\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.499904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.499855    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.499904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.499879    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.600974 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.600874    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.601166 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.600984    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lzn\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-kube-api-access-m2lzn\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.601166 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.601131    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.601166 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.601157    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.601754 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.601732    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.601754 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.601745    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.603671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.603651    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.621898 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.621871    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lzn\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-kube-api-access-m2lzn\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:22.734950 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:22.734920    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:23.041534 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:23.041507    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"]
Oct 30 17:05:23.044601 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:05:23.044567    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6534c29d_0cbf_4097_bb9f_d76fba1c5c83.slice/crio-264b1e338a66c517da12828d45d67879b117a7a401c215a80d8ab3deb5f166da WatchSource:0}: Error finding container 264b1e338a66c517da12828d45d67879b117a7a401c215a80d8ab3deb5f166da: Status 404 returned error can't find the container with id 264b1e338a66c517da12828d45d67879b117a7a401c215a80d8ab3deb5f166da
Oct 30 17:05:23.356115 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:23.356019    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx" event={"ID":"6534c29d-0cbf-4097-bb9f-d76fba1c5c83","Type":"ContainerStarted","Data":"6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103"}
Oct 30 17:05:23.356115 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:23.356082    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx" event={"ID":"6534c29d-0cbf-4097-bb9f-d76fba1c5c83","Type":"ContainerStarted","Data":"264b1e338a66c517da12828d45d67879b117a7a401c215a80d8ab3deb5f166da"}
Oct 30 17:05:23.356612 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:23.356233    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:23.395761 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:23.395709    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx" podStartSLOduration=1.395693319 podStartE2EDuration="1.395693319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:23.394205827 +0000 UTC m=+2553.248084086" watchObservedRunningTime="2026-10-30 17:05:23.395693319 +0000 UTC m=+2553.249571617"
Oct 30 17:05:26.339047 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.339005    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f" podUID="5f8be386-88b3-4154-a086-b20c722b7c3b" containerName="payload-processing" containerID="cri-o://3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d" gracePeriod=25
Oct 30 17:05:26.608274 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.608209    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:05:26.668287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.668255    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-plugins-config-volume\") pod \"5f8be386-88b3-4154-a086-b20c722b7c3b\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") "
Oct 30 17:05:26.668462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.668337    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-routing-overlay-volume\") pod \"5f8be386-88b3-4154-a086-b20c722b7c3b\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") "
Oct 30 17:05:26.668462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.668369    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-provider-credentials\") pod \"5f8be386-88b3-4154-a086-b20c722b7c3b\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") "
Oct 30 17:05:26.668462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.668395    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-kube-api-access-dzwcc\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-kube-api-access-dzwcc\") pod \"5f8be386-88b3-4154-a086-b20c722b7c3b\" (UID: \"5f8be386-88b3-4154-a086-b20c722b7c3b\") "
Oct 30 17:05:26.668731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.668702    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-routing-overlay-volume" pod "5f8be386-88b3-4154-a086-b20c722b7c3b" (UID: "5f8be386-88b3-4154-a086-b20c722b7c3b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:26.668731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.668723    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-plugins-config-volume" pod "5f8be386-88b3-4154-a086-b20c722b7c3b" (UID: "5f8be386-88b3-4154-a086-b20c722b7c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:26.669065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.669047    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:26.669126 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.669071    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f8be386-88b3-4154-a086-b20c722b7c3b-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:26.671311 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.671286    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-kube-api-access-dzwcc" pod "5f8be386-88b3-4154-a086-b20c722b7c3b" (UID: "5f8be386-88b3-4154-a086-b20c722b7c3b"). InnerVolumeSpecName "kube-api-access-dzwcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:26.671438 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.671318    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-provider-credentials" pod "5f8be386-88b3-4154-a086-b20c722b7c3b" (UID: "5f8be386-88b3-4154-a086-b20c722b7c3b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:26.770563 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.770532    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:26.770563 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:26.770561    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwcc\" (UniqueName: \"kubernetes.io/projected/5f8be386-88b3-4154-a086-b20c722b7c3b-kube-api-access-dzwcc\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:27.399179 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.399139    4497 generic.go:356] "Generic (PLEG): container finished" podID="5f8be386-88b3-4154-a086-b20c722b7c3b" containerID="3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d" exitCode=0
Oct 30 17:05:27.399731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.399234    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"
Oct 30 17:05:27.399731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.399374    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f" event={"ID":"5f8be386-88b3-4154-a086-b20c722b7c3b","Type":"ContainerDied","Data":"3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d"}
Oct 30 17:05:27.399731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.399431    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f" event={"ID":"5f8be386-88b3-4154-a086-b20c722b7c3b","Type":"ContainerDied","Data":"feca49a5d212192767c9db3a9298dea015ed92899d02e54075510d7ba8cf4422"}
Oct 30 17:05:27.399731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.399462    4497 scope.go:122] "RemoveContainer" containerID="3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d"
Oct 30 17:05:27.411816 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.411786    4497 scope.go:122] "RemoveContainer" containerID="3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d"
Oct 30 17:05:27.412164 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:05:27.412139    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d\": container with ID starting with 3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d not found: ID does not exist" containerID="3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d"
Oct 30 17:05:27.412277 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.412178    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d"} err="failed to get container status \"3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d\": rpc error: code = NotFound desc = could not find container \"3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d\": container with ID starting with 3c089f9f5e04c13c1c97070a09d3eb3391f2ed30ffdca2696f430cbbe30b4b8d not found: ID does not exist"
Oct 30 17:05:27.444674 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.444642    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"]
Oct 30 17:05:27.454332 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:27.454305    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5976qbp7f"]
Oct 30 17:05:28.300639 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:28.300608    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f8be386-88b3-4154-a086-b20c722b7c3b" path="/var/lib/kubelet/pods/5f8be386-88b3-4154-a086-b20c722b7c3b/volumes"
Oct 30 17:05:31.683353 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.683318    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"]
Oct 30 17:05:31.684304 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.684280    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5f8be386-88b3-4154-a086-b20c722b7c3b" containerName="payload-processing"
Oct 30 17:05:31.684304 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.684302    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8be386-88b3-4154-a086-b20c722b7c3b" containerName="payload-processing"
Oct 30 17:05:31.684504 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.684441    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8be386-88b3-4154-a086-b20c722b7c3b" containerName="payload-processing"
Oct 30 17:05:31.689521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.689498    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:31.703590 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.703564    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-43197b\""
Oct 30 17:05:31.707567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.707532    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"]
Oct 30 17:05:31.713087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.713061    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-43197b-dockercfg-9tjdv\""
Oct 30 17:05:31.730476 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.730442    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"]
Oct 30 17:05:31.734936 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.734904    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:31.738335 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.738305    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc2710-b820-4b0e-92b2-ccf7022bacd7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5\" (UID: \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:31.738582 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.738555    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4pc\" (UniqueName: \"kubernetes.io/projected/74dc2710-b820-4b0e-92b2-ccf7022bacd7-kube-api-access-gx4pc\") pod \"payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5\" (UID: \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:31.753705 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.753674    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"]
Oct 30 17:05:31.854210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.844143    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc2710-b820-4b0e-92b2-ccf7022bacd7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5\" (UID: \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:31.854210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.844384    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4pc\" (UniqueName: \"kubernetes.io/projected/74dc2710-b820-4b0e-92b2-ccf7022bacd7-kube-api-access-gx4pc\") pod \"payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5\" (UID: \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:31.854210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.845653    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc2710-b820-4b0e-92b2-ccf7022bacd7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5\" (UID: \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:31.908473 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.908444    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4pc\" (UniqueName: \"kubernetes.io/projected/74dc2710-b820-4b0e-92b2-ccf7022bacd7-kube-api-access-gx4pc\") pod \"payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5\" (UID: \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:31.946159 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.946074    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gw84\" (UniqueName: \"kubernetes.io/projected/67793c9e-0130-43f7-817e-839d8dd97642-kube-api-access-4gw84\") pod \"payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq\" (UID: \"67793c9e-0130-43f7-817e-839d8dd97642\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:31.946316 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:31.946270    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67793c9e-0130-43f7-817e-839d8dd97642-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq\" (UID: \"67793c9e-0130-43f7-817e-839d8dd97642\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:32.004170 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.004117    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:32.048068 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.048033    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gw84\" (UniqueName: \"kubernetes.io/projected/67793c9e-0130-43f7-817e-839d8dd97642-kube-api-access-4gw84\") pod \"payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq\" (UID: \"67793c9e-0130-43f7-817e-839d8dd97642\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:32.048266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.048181    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67793c9e-0130-43f7-817e-839d8dd97642-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq\" (UID: \"67793c9e-0130-43f7-817e-839d8dd97642\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:32.049055 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.049027    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67793c9e-0130-43f7-817e-839d8dd97642-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq\" (UID: \"67793c9e-0130-43f7-817e-839d8dd97642\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:32.068809 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.068779    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gw84\" (UniqueName: \"kubernetes.io/projected/67793c9e-0130-43f7-817e-839d8dd97642-kube-api-access-4gw84\") pod \"payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq\" (UID: \"67793c9e-0130-43f7-817e-839d8dd97642\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:32.311265 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.311167    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"]
Oct 30 17:05:32.314106 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:05:32.314078    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dc2710_b820_4b0e_92b2_ccf7022bacd7.slice/crio-43e21598f6ff3069681cb444ab6ea29b9cc7359d2f839078682483f8801918af WatchSource:0}: Error finding container 43e21598f6ff3069681cb444ab6ea29b9cc7359d2f839078682483f8801918af: Status 404 returned error can't find the container with id 43e21598f6ff3069681cb444ab6ea29b9cc7359d2f839078682483f8801918af
Oct 30 17:05:32.378446 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.357717    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:32.486000 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.482949    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5" event={"ID":"74dc2710-b820-4b0e-92b2-ccf7022bacd7","Type":"ContainerStarted","Data":"43e21598f6ff3069681cb444ab6ea29b9cc7359d2f839078682483f8801918af"}
Oct 30 17:05:32.790753 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:32.790734    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"]
Oct 30 17:05:32.793483 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:05:32.793456    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67793c9e_0130_43f7_817e_839d8dd97642.slice/crio-092f628b8bd527150935a0c6e5fc6e55762849200291d69ffd0cc62d211dd365 WatchSource:0}: Error finding container 092f628b8bd527150935a0c6e5fc6e55762849200291d69ffd0cc62d211dd365: Status 404 returned error can't find the container with id 092f628b8bd527150935a0c6e5fc6e55762849200291d69ffd0cc62d211dd365
Oct 30 17:05:33.494589 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:33.494222    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5" event={"ID":"74dc2710-b820-4b0e-92b2-ccf7022bacd7","Type":"ContainerStarted","Data":"502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af"}
Oct 30 17:05:33.495423 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:33.495197    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:33.502760 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:33.500352    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq" event={"ID":"67793c9e-0130-43f7-817e-839d8dd97642","Type":"ContainerStarted","Data":"c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0"}
Oct 30 17:05:33.502760 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:33.500422    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq" event={"ID":"67793c9e-0130-43f7-817e-839d8dd97642","Type":"ContainerStarted","Data":"092f628b8bd527150935a0c6e5fc6e55762849200291d69ffd0cc62d211dd365"}
Oct 30 17:05:33.502760 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:33.501474    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:33.531937 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:33.531878    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5" podStartSLOduration=2.531861834 podStartE2EDuration="2.531861834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:33.530938328 +0000 UTC m=+2563.384816594" watchObservedRunningTime="2026-10-30 17:05:33.531861834 +0000 UTC m=+2563.385740101"
Oct 30 17:05:33.568801 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:33.568742    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq" podStartSLOduration=2.568725656 podStartE2EDuration="2.568725656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:33.565688148 +0000 UTC m=+2563.419566413" watchObservedRunningTime="2026-10-30 17:05:33.568725656 +0000 UTC m=+2563.422603920"
Oct 30 17:05:34.376969 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:34.376937    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:05:34.444859 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:34.444822    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"]
Oct 30 17:05:39.445747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.445686    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn" podUID="b23fc42f-30f9-4676-8394-31f2f41b8638" containerName="payload-processing" containerID="cri-o://dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946" gracePeriod=25
Oct 30 17:05:39.724428 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.724395    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:39.779008 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.778978    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-kube-api-access-cgr8w\") pod \"b23fc42f-30f9-4676-8394-31f2f41b8638\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") "
Oct 30 17:05:39.779230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.779037    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-routing-overlay-volume\") pod \"b23fc42f-30f9-4676-8394-31f2f41b8638\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") "
Oct 30 17:05:39.779230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.779161    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-provider-credentials\") pod \"b23fc42f-30f9-4676-8394-31f2f41b8638\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") "
Oct 30 17:05:39.779388 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.779365    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-plugins-config-volume\") pod \"b23fc42f-30f9-4676-8394-31f2f41b8638\" (UID: \"b23fc42f-30f9-4676-8394-31f2f41b8638\") "
Oct 30 17:05:39.779505 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.779475    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-routing-overlay-volume" pod "b23fc42f-30f9-4676-8394-31f2f41b8638" (UID: "b23fc42f-30f9-4676-8394-31f2f41b8638"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:39.779960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.779929    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-plugins-config-volume" pod "b23fc42f-30f9-4676-8394-31f2f41b8638" (UID: "b23fc42f-30f9-4676-8394-31f2f41b8638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:05:39.780103 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.780027    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:39.780103 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.780048    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b23fc42f-30f9-4676-8394-31f2f41b8638-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:39.781676 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.781641    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-kube-api-access-cgr8w" pod "b23fc42f-30f9-4676-8394-31f2f41b8638" (UID: "b23fc42f-30f9-4676-8394-31f2f41b8638"). InnerVolumeSpecName "kube-api-access-cgr8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:39.781885 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.781861    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-provider-credentials" pod "b23fc42f-30f9-4676-8394-31f2f41b8638" (UID: "b23fc42f-30f9-4676-8394-31f2f41b8638"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:05:39.882691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.882649    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgr8w\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-kube-api-access-cgr8w\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:39.882691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:39.882691    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b23fc42f-30f9-4676-8394-31f2f41b8638-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:05:40.554779 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.554741    4497 generic.go:356] "Generic (PLEG): container finished" podID="b23fc42f-30f9-4676-8394-31f2f41b8638" containerID="dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946" exitCode=0
Oct 30 17:05:40.555242 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.554785    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"
Oct 30 17:05:40.555242 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.554831    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn" event={"ID":"b23fc42f-30f9-4676-8394-31f2f41b8638","Type":"ContainerDied","Data":"dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946"}
Oct 30 17:05:40.555242 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.554883    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn" event={"ID":"b23fc42f-30f9-4676-8394-31f2f41b8638","Type":"ContainerDied","Data":"593af00af75745083f6a57f5609d5ffeaa1fb9898a1643e22ffeec8d5843063b"}
Oct 30 17:05:40.555242 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.554908    4497 scope.go:122] "RemoveContainer" containerID="dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946"
Oct 30 17:05:40.564845 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.564829    4497 scope.go:122] "RemoveContainer" containerID="dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946"
Oct 30 17:05:40.565145 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:05:40.565123    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946\": container with ID starting with dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946 not found: ID does not exist" containerID="dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946"
Oct 30 17:05:40.565218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.565154    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946"} err="failed to get container status \"dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946\": rpc error: code = NotFound desc = could not find container \"dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946\": container with ID starting with dcc21c3d7affb7e2b47aae96a89bbe661aa367fed2b0a83c4d1d5482a8d62946 not found: ID does not exist"
Oct 30 17:05:40.600125 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.600089    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"]
Oct 30 17:05:40.609472 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:40.609447    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbc9fwfn"]
Oct 30 17:05:42.301056 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:42.301021    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b23fc42f-30f9-4676-8394-31f2f41b8638" path="/var/lib/kubelet/pods/b23fc42f-30f9-4676-8394-31f2f41b8638/volumes"
Oct 30 17:05:45.516682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:45.516654    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:05:45.517229 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:45.516760    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:05:51.918978 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:05:51.918934    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:05:52.314315 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.314278    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"]
Oct 30 17:05:52.315014 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.314994    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b23fc42f-30f9-4676-8394-31f2f41b8638" containerName="payload-processing"
Oct 30 17:05:52.315014 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.315014    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23fc42f-30f9-4676-8394-31f2f41b8638" containerName="payload-processing"
Oct 30 17:05:52.315283 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.315249    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23fc42f-30f9-4676-8394-31f2f41b8638" containerName="payload-processing"
Oct 30 17:05:52.320623 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.320601    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.344997 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.344965    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"kube-root-ca.crt\""
Oct 30 17:05:52.344997 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.344991    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-plugins-e2e-extmod-b-13c495\""
Oct 30 17:05:52.345241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.345002    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"agc-cred-4bf9743d880e\""
Oct 30 17:05:52.345241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.344963    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-e2e--be0deb1b-dockercfg-vxb7r\""
Oct 30 17:05:52.345241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.344976    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"openshift-service-ca.crt\""
Oct 30 17:05:52.345241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.345059    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"routing-overlay\""
Oct 30 17:05:52.348907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.348881    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"]
Oct 30 17:05:52.376147 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.376117    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.376337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.376163    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.376418 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.376327    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.376814 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.376757    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p4t4\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-kube-api-access-9p4t4\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.477879 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.477844    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p4t4\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-kube-api-access-9p4t4\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.478091 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.477899    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.478091 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.477929    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.478091 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.477973    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.478674 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.478644    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.478792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.478774    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.480531 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.480510    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.498230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.498175    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p4t4\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-kube-api-access-9p4t4\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.634768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.634682    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:52.945540 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:52.945448    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"]
Oct 30 17:05:52.949231 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:05:52.949199    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7d09a3_aa93_4d44_b367_d452e6a8ce9e.slice/crio-474364d1a4e8105b8fb939c12db552225b27568923577553d1212e7dcff5f6d0 WatchSource:0}: Error finding container 474364d1a4e8105b8fb939c12db552225b27568923577553d1212e7dcff5f6d0: Status 404 returned error can't find the container with id 474364d1a4e8105b8fb939c12db552225b27568923577553d1212e7dcff5f6d0
Oct 30 17:05:53.680773 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:53.680732    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n" event={"ID":"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e","Type":"ContainerStarted","Data":"e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4"}
Oct 30 17:05:53.680981 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:53.680807    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n" event={"ID":"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e","Type":"ContainerStarted","Data":"474364d1a4e8105b8fb939c12db552225b27568923577553d1212e7dcff5f6d0"}
Oct 30 17:05:53.680981 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:53.680957    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:05:53.719266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:05:53.719213    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n" podStartSLOduration=1.719198206 podStartE2EDuration="1.719198206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:53.717589489 +0000 UTC m=+2583.571467753" watchObservedRunningTime="2026-10-30 17:05:53.719198206 +0000 UTC m=+2583.573076462"
Oct 30 17:06:04.699002 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:04.698969    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:06:06.816248 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:06.816210    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"]
Oct 30 17:06:11.816508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:11.816459    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx" podUID="6534c29d-0cbf-4097-bb9f-d76fba1c5c83" containerName="payload-processing" containerID="cri-o://6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103" gracePeriod=25
Oct 30 17:06:12.119431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.119402    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:06:12.177551 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.177511    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-plugins-config-volume\") pod \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") "
Oct 30 17:06:12.177738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.177599    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-provider-credentials\") pod \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") "
Oct 30 17:06:12.177738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.177721    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-routing-overlay-volume\") pod \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") "
Oct 30 17:06:12.177916 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.177899    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-kube-api-access-m2lzn\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-kube-api-access-m2lzn\") pod \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\" (UID: \"6534c29d-0cbf-4097-bb9f-d76fba1c5c83\") "
Oct 30 17:06:12.178295 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.177989    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-plugins-config-volume" pod "6534c29d-0cbf-4097-bb9f-d76fba1c5c83" (UID: "6534c29d-0cbf-4097-bb9f-d76fba1c5c83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:06:12.178295 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.178178    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-routing-overlay-volume" pod "6534c29d-0cbf-4097-bb9f-d76fba1c5c83" (UID: "6534c29d-0cbf-4097-bb9f-d76fba1c5c83"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:06:12.178556 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.178530    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:12.178556 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.178554    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:12.180323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.180280    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-provider-credentials" pod "6534c29d-0cbf-4097-bb9f-d76fba1c5c83" (UID: "6534c29d-0cbf-4097-bb9f-d76fba1c5c83"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:06:12.180592 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.180566    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-kube-api-access-m2lzn" pod "6534c29d-0cbf-4097-bb9f-d76fba1c5c83" (UID: "6534c29d-0cbf-4097-bb9f-d76fba1c5c83"). InnerVolumeSpecName "kube-api-access-m2lzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:06:12.280136 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.280101    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2lzn\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-kube-api-access-m2lzn\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:12.280136 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.280133    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6534c29d-0cbf-4097-bb9f-d76fba1c5c83-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:12.817289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.817248    4497 generic.go:356] "Generic (PLEG): container finished" podID="6534c29d-0cbf-4097-bb9f-d76fba1c5c83" containerID="6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103" exitCode=0
Oct 30 17:06:12.817781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.817296    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"
Oct 30 17:06:12.817781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.817333    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx" event={"ID":"6534c29d-0cbf-4097-bb9f-d76fba1c5c83","Type":"ContainerDied","Data":"6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103"}
Oct 30 17:06:12.817781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.817391    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx" event={"ID":"6534c29d-0cbf-4097-bb9f-d76fba1c5c83","Type":"ContainerDied","Data":"264b1e338a66c517da12828d45d67879b117a7a401c215a80d8ab3deb5f166da"}
Oct 30 17:06:12.817781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.817421    4497 scope.go:122] "RemoveContainer" containerID="6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103"
Oct 30 17:06:12.827965 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.827946    4497 scope.go:122] "RemoveContainer" containerID="6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103"
Oct 30 17:06:12.828256 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:06:12.828233    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103\": container with ID starting with 6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103 not found: ID does not exist" containerID="6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103"
Oct 30 17:06:12.828318 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.828266    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103"} err="failed to get container status \"6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103\": rpc error: code = NotFound desc = could not find container \"6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103\": container with ID starting with 6b3f0b0dc735932032f8f47900fbc3849269d48dd970c9d40068789d154ca103 not found: ID does not exist"
Oct 30 17:06:12.862479 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.862446    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"]
Oct 30 17:06:12.871662 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:12.871635    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-5d49s4fhx"]
Oct 30 17:06:14.301453 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:14.301421    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6534c29d-0cbf-4097-bb9f-d76fba1c5c83" path="/var/lib/kubelet/pods/6534c29d-0cbf-4097-bb9f-d76fba1c5c83/volumes"
Oct 30 17:06:21.924543 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:06:21.924508    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:06:23.686394 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.686360    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"]
Oct 30 17:06:23.687057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.687043    4497 cpu_manager.go:414] "containerMap: removing container" podUID="6534c29d-0cbf-4097-bb9f-d76fba1c5c83" containerName="payload-processing"
Oct 30 17:06:23.687102 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.687058    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6534c29d-0cbf-4097-bb9f-d76fba1c5c83" containerName="payload-processing"
Oct 30 17:06:23.687222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.687180    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="6534c29d-0cbf-4097-bb9f-d76fba1c5c83" containerName="payload-processing"
Oct 30 17:06:23.690856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.690835    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:23.706335 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.706312    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"e2e-cmp-legacy-9e3cf0-provider-cred\""
Oct 30 17:06:23.713635 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.713612    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"routing-overlay\""
Oct 30 17:06:23.713782 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.713627    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d5b798\""
Oct 30 17:06:23.713876 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.713742    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-e2e--02a9e3c4-dockercfg-cd9xl\""
Oct 30 17:06:23.716032 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.716008    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"kube-root-ca.crt\""
Oct 30 17:06:23.716267 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.716247    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"openshift-service-ca.crt\""
Oct 30 17:06:23.718648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.718622    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"]
Oct 30 17:06:23.902425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.902394    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"]
Oct 30 17:06:23.909931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.908560    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:23.909931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.908824    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qjzd\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-kube-api-access-4qjzd\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:23.909931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.908966    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:23.909931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:23.909031    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.010277 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.010222    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.010482 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.010291    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.010482 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.010350    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.010482 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.010475    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qjzd\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-kube-api-access-4qjzd\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.011106 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.011081    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.011273 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.011228    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.012806 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.012790    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.031786 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.031752    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qjzd\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-kube-api-access-4qjzd\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-54987744p\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.306107 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.306016    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.629132 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.628956    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"]
Oct 30 17:06:24.631921 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:06:24.631899    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5da0459_0be2_493c_a1ae_57e640e5b336.slice/crio-02769bebd66c3294ce197ff088cfb1b1451b48fd4a6730d9cea6b89377b85d85 WatchSource:0}: Error finding container 02769bebd66c3294ce197ff088cfb1b1451b48fd4a6730d9cea6b89377b85d85: Status 404 returned error can't find the container with id 02769bebd66c3294ce197ff088cfb1b1451b48fd4a6730d9cea6b89377b85d85
Oct 30 17:06:24.913784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.913697    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p" event={"ID":"b5da0459-0be2-493c-a1ae-57e640e5b336","Type":"ContainerStarted","Data":"7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3"}
Oct 30 17:06:24.913784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.913761    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p" event={"ID":"b5da0459-0be2-493c-a1ae-57e640e5b336","Type":"ContainerStarted","Data":"02769bebd66c3294ce197ff088cfb1b1451b48fd4a6730d9cea6b89377b85d85"}
Oct 30 17:06:24.913784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.913797    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:24.952791 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:24.952732    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p" podStartSLOduration=1.9527148699999999 podStartE2EDuration="1.95271487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:24.950406721 +0000 UTC m=+2614.804284989" watchObservedRunningTime="2026-10-30 17:06:24.95271487 +0000 UTC m=+2614.806593136"
Oct 30 17:06:28.903772 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:28.903722    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n" podUID="6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" containerName="payload-processing" containerID="cri-o://e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4" gracePeriod=25
Oct 30 17:06:29.192109 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.192081    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:06:29.305721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.305681    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-provider-credentials\") pod \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") "
Oct 30 17:06:29.305910 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.305862    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-routing-overlay-volume\") pod \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") "
Oct 30 17:06:29.305910 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.305907    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-kube-api-access-9p4t4\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-kube-api-access-9p4t4\") pod \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") "
Oct 30 17:06:29.306031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.305981    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-plugins-config-volume\") pod \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\" (UID: \"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e\") "
Oct 30 17:06:29.306372 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.306272    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-routing-overlay-volume" pod "6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" (UID: "6c7d09a3-aa93-4d44-b367-d452e6a8ce9e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:06:29.306499 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.306465    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-plugins-config-volume" pod "6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" (UID: "6c7d09a3-aa93-4d44-b367-d452e6a8ce9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:06:29.307951 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.307927    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-provider-credentials" pod "6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" (UID: "6c7d09a3-aa93-4d44-b367-d452e6a8ce9e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:06:29.308064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.308049    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-kube-api-access-9p4t4" pod "6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" (UID: "6c7d09a3-aa93-4d44-b367-d452e6a8ce9e"). InnerVolumeSpecName "kube-api-access-9p4t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:06:29.407536 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.407427    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:29.407738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.407572    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:29.407738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.407602    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p4t4\" (UniqueName: \"kubernetes.io/projected/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-kube-api-access-9p4t4\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:29.407738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.407615    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:29.950172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.950134    4497 generic.go:356] "Generic (PLEG): container finished" podID="6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" containerID="e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4" exitCode=0
Oct 30 17:06:29.950633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.950178    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"
Oct 30 17:06:29.950633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.950229    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n" event={"ID":"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e","Type":"ContainerDied","Data":"e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4"}
Oct 30 17:06:29.950633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.950292    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n" event={"ID":"6c7d09a3-aa93-4d44-b367-d452e6a8ce9e","Type":"ContainerDied","Data":"474364d1a4e8105b8fb939c12db552225b27568923577553d1212e7dcff5f6d0"}
Oct 30 17:06:29.950633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.950324    4497 scope.go:122] "RemoveContainer" containerID="e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4"
Oct 30 17:06:29.961941 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.961919    4497 scope.go:122] "RemoveContainer" containerID="e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4"
Oct 30 17:06:29.962264 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:06:29.962239    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4\": container with ID starting with e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4 not found: ID does not exist" containerID="e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4"
Oct 30 17:06:29.962339 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.962272    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4"} err="failed to get container status \"e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4\": rpc error: code = NotFound desc = could not find container \"e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4\": container with ID starting with e3ab4ce548bf045373edbc0efd014f2d194c5ebc0f9af860dd630c93bba445f4 not found: ID does not exist"
Oct 30 17:06:29.998034 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:29.998000    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"]
Oct 30 17:06:30.006382 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:30.006352    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44g4s4n"]
Oct 30 17:06:30.302087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:30.302056    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" path="/var/lib/kubelet/pods/6c7d09a3-aa93-4d44-b367-d452e6a8ce9e/volumes"
Oct 30 17:06:35.924145 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:35.924115    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:06:38.411511 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.411472    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"]
Oct 30 17:06:38.412222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.412174    4497 cpu_manager.go:414] "containerMap: removing container" podUID="6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" containerName="payload-processing"
Oct 30 17:06:38.412222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.412200    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" containerName="payload-processing"
Oct 30 17:06:38.412334 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.412328    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7d09a3-aa93-4d44-b367-d452e6a8ce9e" containerName="payload-processing"
Oct 30 17:06:38.417369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.417353    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.451989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.451947    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"]
Oct 30 17:06:38.452431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.452404    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"routing-overlay\""
Oct 30 17:06:38.452545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.452482    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-e2e--22b3ecb4-dockercfg-6tsff\""
Oct 30 17:06:38.452545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.452523    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-3e1593902499\""
Oct 30 17:06:38.452883 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.452861    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"openshift-service-ca.crt\""
Oct 30 17:06:38.452991 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.452961    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"kube-root-ca.crt\""
Oct 30 17:06:38.453064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.453020    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5c6b3c\""
Oct 30 17:06:38.571182 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.571146    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.571376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.571358    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.571504 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.571486    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67ns\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-kube-api-access-w67ns\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.571560 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.571543    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.673292 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.673205    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.673466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.673301    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67ns\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-kube-api-access-w67ns\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.673466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.673352    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.673466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.673419    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.674103 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.674075    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.674319 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.674293    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.675821 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.675800    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.695056 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.695030    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67ns\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-kube-api-access-w67ns\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:38.731507 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:38.731477    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:39.041222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:39.041199    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"]
Oct 30 17:06:39.043742 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:06:39.043714    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d8d567_4384_47b5_a686_83d7be3ff2ac.slice/crio-79eb99937d3179f22963374850b2aeb70f429592e415c20e143aedd94474481d WatchSource:0}: Error finding container 79eb99937d3179f22963374850b2aeb70f429592e415c20e143aedd94474481d: Status 404 returned error can't find the container with id 79eb99937d3179f22963374850b2aeb70f429592e415c20e143aedd94474481d
Oct 30 17:06:40.049196 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:40.049157    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn" event={"ID":"33d8d567-4384-47b5-a686-83d7be3ff2ac","Type":"ContainerStarted","Data":"634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2"}
Oct 30 17:06:40.049613 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:40.049253    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:40.049613 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:40.049272    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn" event={"ID":"33d8d567-4384-47b5-a686-83d7be3ff2ac","Type":"ContainerStarted","Data":"79eb99937d3179f22963374850b2aeb70f429592e415c20e143aedd94474481d"}
Oct 30 17:06:40.087674 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:40.087621    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn" podStartSLOduration=2.087606755 podStartE2EDuration="2.087606755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:40.086240188 +0000 UTC m=+2629.940118454" watchObservedRunningTime="2026-10-30 17:06:40.087606755 +0000 UTC m=+2629.941485026"
Oct 30 17:06:49.356740 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:49.356697    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"]
Oct 30 17:06:49.360302 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:49.360277    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:51.513235 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:51.513181    4497 scope.go:122] "RemoveContainer" containerID="18c879a7a59a595c62e894c4637deb0bed1a8daa2d10508e0d6c936fa5c823b5"
Oct 30 17:06:51.920087 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:06:51.920005    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:06:54.357328 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.357288    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn" podUID="33d8d567-4384-47b5-a686-83d7be3ff2ac" containerName="payload-processing" containerID="cri-o://634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2" gracePeriod=25
Oct 30 17:06:54.637466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.637434    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:54.780313 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.780274    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-provider-credentials\") pod \"33d8d567-4384-47b5-a686-83d7be3ff2ac\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") "
Oct 30 17:06:54.780492 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.780322    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-plugins-config-volume\") pod \"33d8d567-4384-47b5-a686-83d7be3ff2ac\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") "
Oct 30 17:06:54.780492 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.780452    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-kube-api-access-w67ns\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-kube-api-access-w67ns\") pod \"33d8d567-4384-47b5-a686-83d7be3ff2ac\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") "
Oct 30 17:06:54.780602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.780585    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-routing-overlay-volume\") pod \"33d8d567-4384-47b5-a686-83d7be3ff2ac\" (UID: \"33d8d567-4384-47b5-a686-83d7be3ff2ac\") "
Oct 30 17:06:54.780894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.780867    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-plugins-config-volume" pod "33d8d567-4384-47b5-a686-83d7be3ff2ac" (UID: "33d8d567-4384-47b5-a686-83d7be3ff2ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:06:54.781019 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.780941    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-routing-overlay-volume" pod "33d8d567-4384-47b5-a686-83d7be3ff2ac" (UID: "33d8d567-4384-47b5-a686-83d7be3ff2ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:06:54.781288 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.781259    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:54.781366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.781292    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33d8d567-4384-47b5-a686-83d7be3ff2ac-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:54.782655 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.782631    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-kube-api-access-w67ns" pod "33d8d567-4384-47b5-a686-83d7be3ff2ac" (UID: "33d8d567-4384-47b5-a686-83d7be3ff2ac"). InnerVolumeSpecName "kube-api-access-w67ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:06:54.782750 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.782733    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-provider-credentials" pod "33d8d567-4384-47b5-a686-83d7be3ff2ac" (UID: "33d8d567-4384-47b5-a686-83d7be3ff2ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:06:54.882229 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.882118    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:54.882229 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:54.882150    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67ns\" (UniqueName: \"kubernetes.io/projected/33d8d567-4384-47b5-a686-83d7be3ff2ac-kube-api-access-w67ns\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:06:55.168868 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.168775    4497 generic.go:356] "Generic (PLEG): container finished" podID="33d8d567-4384-47b5-a686-83d7be3ff2ac" containerID="634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2" exitCode=0
Oct 30 17:06:55.168868 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.168825    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"
Oct 30 17:06:55.169076 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.168862    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn" event={"ID":"33d8d567-4384-47b5-a686-83d7be3ff2ac","Type":"ContainerDied","Data":"634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2"}
Oct 30 17:06:55.169076 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.168945    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn" event={"ID":"33d8d567-4384-47b5-a686-83d7be3ff2ac","Type":"ContainerDied","Data":"79eb99937d3179f22963374850b2aeb70f429592e415c20e143aedd94474481d"}
Oct 30 17:06:55.169076 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.168968    4497 scope.go:122] "RemoveContainer" containerID="634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2"
Oct 30 17:06:55.184175 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.184151    4497 scope.go:122] "RemoveContainer" containerID="634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2"
Oct 30 17:06:55.184737 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:06:55.184713    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2\": container with ID starting with 634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2 not found: ID does not exist" containerID="634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2"
Oct 30 17:06:55.184831 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.184750    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2"} err="failed to get container status \"634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2\": rpc error: code = NotFound desc = could not find container \"634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2\": container with ID starting with 634268c69ecdbb87aab8cc5099c203933cb406cfa06b3d8117c81a38c2e84ab2 not found: ID does not exist"
Oct 30 17:06:55.216756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.216724    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"]
Oct 30 17:06:55.221863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:55.221835    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d6l24hn"]
Oct 30 17:06:56.300931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:06:56.300895    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d8d567-4384-47b5-a686-83d7be3ff2ac" path="/var/lib/kubelet/pods/33d8d567-4384-47b5-a686-83d7be3ff2ac/volumes"
Oct 30 17:07:07.283870 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.283780    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"]
Oct 30 17:07:07.284541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.284525    4497 cpu_manager.go:414] "containerMap: removing container" podUID="33d8d567-4384-47b5-a686-83d7be3ff2ac" containerName="payload-processing"
Oct 30 17:07:07.284587 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.284542    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d8d567-4384-47b5-a686-83d7be3ff2ac" containerName="payload-processing"
Oct 30 17:07:07.284686 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.284677    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d8d567-4384-47b5-a686-83d7be3ff2ac" containerName="payload-processing"
Oct 30 17:07:07.289689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.289666    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.301746 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.301718    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"agc-cred-c1ddc76883fe\""
Oct 30 17:07:07.311661 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.311633    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"routing-overlay\""
Oct 30 17:07:07.311661 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.311649    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2589b5\""
Oct 30 17:07:07.314321 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.314285    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"openshift-service-ca.crt\""
Oct 30 17:07:07.314494 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.314447    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"kube-root-ca.crt\""
Oct 30 17:07:07.320010 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.319954    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"]
Oct 30 17:07:07.361097 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.361057    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8sd\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-kube-api-access-kd8sd\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.361348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.361266    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.361464 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.361446    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.361636 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.361616    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.404211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.403230    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-e2e--8e3f1132-dockercfg-gjbn7\""
Oct 30 17:07:07.470211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.464363    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.470211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.464463    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8sd\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-kube-api-access-kd8sd\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.470211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.464605    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.470211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.464730    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.470211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.466431    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.470211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.467160    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.474212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.473947    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.498575 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.498537    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8sd\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-kube-api-access-kd8sd\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.605266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.605125    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:07.917791 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:07.917588    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"]
Oct 30 17:07:07.921526 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:07:07.921484    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1725817c_2b43_4edf_a8fd_f7343ef389ce.slice/crio-24f0a02b9a02ffd4436357476fccf3b84f553e34c3dba33c2e2079c9de729092 WatchSource:0}: Error finding container 24f0a02b9a02ffd4436357476fccf3b84f553e34c3dba33c2e2079c9de729092: Status 404 returned error can't find the container with id 24f0a02b9a02ffd4436357476fccf3b84f553e34c3dba33c2e2079c9de729092
Oct 30 17:07:08.302414 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:08.302378    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" event={"ID":"1725817c-2b43-4edf-a8fd-f7343ef389ce","Type":"ContainerStarted","Data":"4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d"}
Oct 30 17:07:08.302932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:08.302435    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" event={"ID":"1725817c-2b43-4edf-a8fd-f7343ef389ce","Type":"ContainerStarted","Data":"24f0a02b9a02ffd4436357476fccf3b84f553e34c3dba33c2e2079c9de729092"}
Oct 30 17:07:08.302932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:08.302467    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:08.340178 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:08.340114    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" podStartSLOduration=1.340098125 podStartE2EDuration="1.340098125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:08.338801304 +0000 UTC m=+2658.192679569" watchObservedRunningTime="2026-10-30 17:07:08.340098125 +0000 UTC m=+2658.193976389"
Oct 30 17:07:09.404836 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:09.404797    4497 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:09.404836 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:09.404829    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9: secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:09.405479 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:09.404912    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials podName:1725817c-2b43-4edf-a8fd-f7343ef389ce nodeName:}" failed. No retries permitted until 2026-10-30 17:07:09.904897786 +0000 UTC m=+2659.758776029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" (UID: "1725817c-2b43-4edf-a8fd-f7343ef389ce") : secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:09.654096 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:09.654063    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"]
Oct 30 17:07:09.913117 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:09.913082    4497 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:09.913117 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:09.913109    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9: secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:09.913352 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:09.913176    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials podName:1725817c-2b43-4edf-a8fd-f7343ef389ce nodeName:}" failed. No retries permitted until 2026-10-30 17:07:10.913160151 +0000 UTC m=+2660.767038411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" (UID: "1725817c-2b43-4edf-a8fd-f7343ef389ce") : secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:10.925937 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:10.925898    4497 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:10.925937 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:10.925922    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9: secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:10.926398 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:10.925972    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials podName:1725817c-2b43-4edf-a8fd-f7343ef389ce nodeName:}" failed. No retries permitted until 2026-10-30 17:07:12.925957772 +0000 UTC m=+2662.779836033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" (UID: "1725817c-2b43-4edf-a8fd-f7343ef389ce") : secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:12.962432 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:12.962395    4497 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:12.962432 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:12.962426    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9: secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:12.963010 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:12.962519    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials podName:1725817c-2b43-4edf-a8fd-f7343ef389ce nodeName:}" failed. No retries permitted until 2026-10-30 17:07:16.962499909 +0000 UTC m=+2666.816378156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" (UID: "1725817c-2b43-4edf-a8fd-f7343ef389ce") : secret "agc-cred-c1ddc76883fe" not found
Oct 30 17:07:15.316164 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.316103    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" podUID="1725817c-2b43-4edf-a8fd-f7343ef389ce" containerName="payload-processing" containerID="cri-o://4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d" gracePeriod=25
Oct 30 17:07:15.592765 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.592736    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:15.609395 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.609363    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials\") pod \"1725817c-2b43-4edf-a8fd-f7343ef389ce\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") "
Oct 30 17:07:15.609545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.609421    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-kube-api-access-kd8sd\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-kube-api-access-kd8sd\") pod \"1725817c-2b43-4edf-a8fd-f7343ef389ce\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") "
Oct 30 17:07:15.609545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.609448    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-plugins-config-volume\") pod \"1725817c-2b43-4edf-a8fd-f7343ef389ce\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") "
Oct 30 17:07:15.609646 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.609623    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-routing-overlay-volume\") pod \"1725817c-2b43-4edf-a8fd-f7343ef389ce\" (UID: \"1725817c-2b43-4edf-a8fd-f7343ef389ce\") "
Oct 30 17:07:15.609934 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.609904    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-plugins-config-volume" pod "1725817c-2b43-4edf-a8fd-f7343ef389ce" (UID: "1725817c-2b43-4edf-a8fd-f7343ef389ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:07:15.610037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.609977    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-routing-overlay-volume" pod "1725817c-2b43-4edf-a8fd-f7343ef389ce" (UID: "1725817c-2b43-4edf-a8fd-f7343ef389ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:07:15.610215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.610200    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:15.610269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.610220    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1725817c-2b43-4edf-a8fd-f7343ef389ce-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:15.611616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.611590    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials" pod "1725817c-2b43-4edf-a8fd-f7343ef389ce" (UID: "1725817c-2b43-4edf-a8fd-f7343ef389ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:15.611708 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.611622    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-kube-api-access-kd8sd" pod "1725817c-2b43-4edf-a8fd-f7343ef389ce" (UID: "1725817c-2b43-4edf-a8fd-f7343ef389ce"). InnerVolumeSpecName "kube-api-access-kd8sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:15.711874 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.711837    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:15.711874 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:15.711867    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd8sd\" (UniqueName: \"kubernetes.io/projected/1725817c-2b43-4edf-a8fd-f7343ef389ce-kube-api-access-kd8sd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:16.423490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.423455    4497 generic.go:356] "Generic (PLEG): container finished" podID="1725817c-2b43-4edf-a8fd-f7343ef389ce" containerID="4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d" exitCode=0
Oct 30 17:07:16.423960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.423499    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"
Oct 30 17:07:16.423960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.423509    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" event={"ID":"1725817c-2b43-4edf-a8fd-f7343ef389ce","Type":"ContainerDied","Data":"4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d"}
Oct 30 17:07:16.423960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.423561    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9" event={"ID":"1725817c-2b43-4edf-a8fd-f7343ef389ce","Type":"ContainerDied","Data":"24f0a02b9a02ffd4436357476fccf3b84f553e34c3dba33c2e2079c9de729092"}
Oct 30 17:07:16.423960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.423592    4497 scope.go:122] "RemoveContainer" containerID="4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d"
Oct 30 17:07:16.433564 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.433539    4497 scope.go:122] "RemoveContainer" containerID="4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d"
Oct 30 17:07:16.433885 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:16.433865    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d\": container with ID starting with 4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d not found: ID does not exist" containerID="4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d"
Oct 30 17:07:16.433946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.433897    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d"} err="failed to get container status \"4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d\": rpc error: code = NotFound desc = could not find container \"4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d\": container with ID starting with 4f41b4f6ce1d6f9c91cf9b1f60feff70a3d26ffbf2636dde0e760778e112b77d not found: ID does not exist"
Oct 30 17:07:16.468589 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.468553    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"]
Oct 30 17:07:16.503020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:16.502978    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbbs8tv9"]
Oct 30 17:07:17.643652 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.643612    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"]
Oct 30 17:07:17.644713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.644692    4497 cpu_manager.go:414] "containerMap: removing container" podUID="1725817c-2b43-4edf-a8fd-f7343ef389ce" containerName="payload-processing"
Oct 30 17:07:17.644812 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.644714    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1725817c-2b43-4edf-a8fd-f7343ef389ce" containerName="payload-processing"
Oct 30 17:07:17.644892 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.644879    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="1725817c-2b43-4edf-a8fd-f7343ef389ce" containerName="payload-processing"
Oct 30 17:07:17.650687 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.650660    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.673393 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.673358    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-129cce3672c2\""
Oct 30 17:07:17.673626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.673488    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-6be2aaced4b8\""
Oct 30 17:07:17.673962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.673745    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"routing-overlay\""
Oct 30 17:07:17.673962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.673825    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"kube-root-ca.crt\""
Oct 30 17:07:17.673962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.673888    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3106d6\""
Oct 30 17:07:17.673962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.673925    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"openshift-service-ca.crt\""
Oct 30 17:07:17.674457 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.674433    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-e2e--ac3a616d-dockercfg-nw26z\""
Oct 30 17:07:17.678833 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.678795    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"]
Oct 30 17:07:17.746595 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.746561    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvhw\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-kube-api-access-jbvhw\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.746805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.746731    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.746805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.746794    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.746951 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.746885    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.848396 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.848359    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvhw\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-kube-api-access-jbvhw\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.848593 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.848482    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.848593 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.848542    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.848720 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.848629    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.849280 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.849255    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.849399 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.849377    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.851112 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.851091    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.868773 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.868751    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvhw\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-kube-api-access-jbvhw\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:17.983262 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:17.981484    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:18.302920 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:18.302870    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1725817c-2b43-4edf-a8fd-f7343ef389ce" path="/var/lib/kubelet/pods/1725817c-2b43-4edf-a8fd-f7343ef389ce/volumes"
Oct 30 17:07:18.441679 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:18.441637    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"]
Oct 30 17:07:18.444505 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:07:18.444477    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b21daf_c132_4d83_b583_eb768e9004ad.slice/crio-2044f3b0b5247f471eb82cc03cf3a640e956c3afa11ced9a74d2a97ba5d1e58d WatchSource:0}: Error finding container 2044f3b0b5247f471eb82cc03cf3a640e956c3afa11ced9a74d2a97ba5d1e58d: Status 404 returned error can't find the container with id 2044f3b0b5247f471eb82cc03cf3a640e956c3afa11ced9a74d2a97ba5d1e58d
Oct 30 17:07:18.501898 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:18.501870    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"]
Oct 30 17:07:19.447249 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:19.447211    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg" event={"ID":"36b21daf-c132-4d83-b583-eb768e9004ad","Type":"ContainerStarted","Data":"87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b"}
Oct 30 17:07:19.447715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:19.447277    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg" event={"ID":"36b21daf-c132-4d83-b583-eb768e9004ad","Type":"ContainerStarted","Data":"2044f3b0b5247f471eb82cc03cf3a640e956c3afa11ced9a74d2a97ba5d1e58d"}
Oct 30 17:07:19.447715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:19.447308    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:19.484749 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:19.484691    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg" podStartSLOduration=2.484675935 podStartE2EDuration="2.484675935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:19.484145779 +0000 UTC m=+2669.338024026" watchObservedRunningTime="2026-10-30 17:07:19.484675935 +0000 UTC m=+2669.338554200"
Oct 30 17:07:21.919491 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:21.919455    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:07:24.447534 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.447489    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg" podUID="36b21daf-c132-4d83-b583-eb768e9004ad" containerName="payload-processing" containerID="cri-o://87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b" gracePeriod=25
Oct 30 17:07:24.727922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.727892    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:24.889648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.889614    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-kube-api-access-jbvhw\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-kube-api-access-jbvhw\") pod \"36b21daf-c132-4d83-b583-eb768e9004ad\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") "
Oct 30 17:07:24.889841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.889779    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-routing-overlay-volume\") pod \"36b21daf-c132-4d83-b583-eb768e9004ad\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") "
Oct 30 17:07:24.889841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.889817    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-provider-credentials\") pod \"36b21daf-c132-4d83-b583-eb768e9004ad\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") "
Oct 30 17:07:24.889933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.889845    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-plugins-config-volume\") pod \"36b21daf-c132-4d83-b583-eb768e9004ad\" (UID: \"36b21daf-c132-4d83-b583-eb768e9004ad\") "
Oct 30 17:07:24.890218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.890178    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-routing-overlay-volume" pod "36b21daf-c132-4d83-b583-eb768e9004ad" (UID: "36b21daf-c132-4d83-b583-eb768e9004ad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:07:24.890331 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.890308    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-plugins-config-volume" pod "36b21daf-c132-4d83-b583-eb768e9004ad" (UID: "36b21daf-c132-4d83-b583-eb768e9004ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:07:24.890490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.890472    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:24.890562 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.890492    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b21daf-c132-4d83-b583-eb768e9004ad-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:24.891951 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.891928    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-kube-api-access-jbvhw" pod "36b21daf-c132-4d83-b583-eb768e9004ad" (UID: "36b21daf-c132-4d83-b583-eb768e9004ad"). InnerVolumeSpecName "kube-api-access-jbvhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:24.892047 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.891956    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-provider-credentials" pod "36b21daf-c132-4d83-b583-eb768e9004ad" (UID: "36b21daf-c132-4d83-b583-eb768e9004ad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:24.992255 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.992133    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbvhw\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-kube-api-access-jbvhw\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:24.992255 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:24.992167    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b21daf-c132-4d83-b583-eb768e9004ad-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:25.493034 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.493003    4497 generic.go:356] "Generic (PLEG): container finished" podID="36b21daf-c132-4d83-b583-eb768e9004ad" containerID="87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b" exitCode=0
Oct 30 17:07:25.493470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.493061    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"
Oct 30 17:07:25.493470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.493085    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg" event={"ID":"36b21daf-c132-4d83-b583-eb768e9004ad","Type":"ContainerDied","Data":"87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b"}
Oct 30 17:07:25.493470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.493147    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg" event={"ID":"36b21daf-c132-4d83-b583-eb768e9004ad","Type":"ContainerDied","Data":"2044f3b0b5247f471eb82cc03cf3a640e956c3afa11ced9a74d2a97ba5d1e58d"}
Oct 30 17:07:25.493470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.493178    4497 scope.go:122] "RemoveContainer" containerID="87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b"
Oct 30 17:07:25.504352 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.504333    4497 scope.go:122] "RemoveContainer" containerID="87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b"
Oct 30 17:07:25.504618 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:25.504601    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b\": container with ID starting with 87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b not found: ID does not exist" containerID="87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b"
Oct 30 17:07:25.504663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.504626    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b"} err="failed to get container status \"87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b\": rpc error: code = NotFound desc = could not find container \"87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b\": container with ID starting with 87f8b54d2f5230da52ea314fea0940c42f000e4d0e9a09fb6a25e08023fd568b not found: ID does not exist"
Oct 30 17:07:25.539363 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.539325    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"]
Oct 30 17:07:25.548715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:25.548689    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785c5h8cg"]
Oct 30 17:07:26.302723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:26.302684    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b21daf-c132-4d83-b583-eb768e9004ad" path="/var/lib/kubelet/pods/36b21daf-c132-4d83-b583-eb768e9004ad/volumes"
Oct 30 17:07:47.819581 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:47.819542    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"]
Oct 30 17:07:51.542021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.541982    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"]
Oct 30 17:07:51.542909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.542886    4497 cpu_manager.go:414] "containerMap: removing container" podUID="36b21daf-c132-4d83-b583-eb768e9004ad" containerName="payload-processing"
Oct 30 17:07:51.543022 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.542910    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b21daf-c132-4d83-b583-eb768e9004ad" containerName="payload-processing"
Oct 30 17:07:51.543140 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.543125    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b21daf-c132-4d83-b583-eb768e9004ad" containerName="payload-processing"
Oct 30 17:07:51.546625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.546605    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.558613 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.558589    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"agc-cred-8dda90d8fcab\""
Oct 30 17:07:51.568031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.568006    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-e2e--02a9e3c4-dockercfg-cqwmp\""
Oct 30 17:07:51.572562 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.572539    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"]
Oct 30 17:07:51.578083 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.578055    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtmb\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-kube-api-access-mgtmb\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.578227 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.578181    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.578312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.578293    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.578378 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.578346    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.699615 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.690230    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtmb\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-kube-api-access-mgtmb\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.699615 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.690360    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.699615 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.690471    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.699615 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.690520    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.699615 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.692316    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.712324 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.712288    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.716378 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.716288    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtmb\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-kube-api-access-mgtmb\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.731257 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.730217    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.861162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:51.861080    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:51.923923 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:51.923868    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:07:52.182959 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:52.182917    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"]
Oct 30 17:07:52.186261 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:07:52.186231    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad55803d_19c6_4156_905e_ab89f0635dc2.slice/crio-cc0d0ec8c8f92c71db5f963eec54204d621eff475ac71628707a34f5cef5f360 WatchSource:0}: Error finding container cc0d0ec8c8f92c71db5f963eec54204d621eff475ac71628707a34f5cef5f360: Status 404 returned error can't find the container with id cc0d0ec8c8f92c71db5f963eec54204d621eff475ac71628707a34f5cef5f360
Oct 30 17:07:52.723630 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:52.723466    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6" event={"ID":"ad55803d-19c6-4156-905e-ab89f0635dc2","Type":"ContainerStarted","Data":"c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb"}
Oct 30 17:07:52.724382 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:52.723840    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6" event={"ID":"ad55803d-19c6-4156-905e-ab89f0635dc2","Type":"ContainerStarted","Data":"cc0d0ec8c8f92c71db5f963eec54204d621eff475ac71628707a34f5cef5f360"}
Oct 30 17:07:52.724382 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:52.723893    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:07:52.762035 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:52.761968    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6" podStartSLOduration=1.761950443 podStartE2EDuration="1.761950443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:52.760009854 +0000 UTC m=+2702.613888119" watchObservedRunningTime="2026-10-30 17:07:52.761950443 +0000 UTC m=+2702.615828708"
Oct 30 17:07:52.819866 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:52.819821    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p" podUID="b5da0459-0be2-493c-a1ae-57e640e5b336" containerName="payload-processing" containerID="cri-o://7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3" gracePeriod=25
Oct 30 17:07:53.087007 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.086976    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:07:53.212516 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.212479    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-plugins-config-volume\") pod \"b5da0459-0be2-493c-a1ae-57e640e5b336\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") "
Oct 30 17:07:53.212671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.212638    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-routing-overlay-volume\") pod \"b5da0459-0be2-493c-a1ae-57e640e5b336\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") "
Oct 30 17:07:53.212730 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.212719    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-provider-credentials\") pod \"b5da0459-0be2-493c-a1ae-57e640e5b336\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") "
Oct 30 17:07:53.212785 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.212767    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-kube-api-access-4qjzd\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-kube-api-access-4qjzd\") pod \"b5da0459-0be2-493c-a1ae-57e640e5b336\" (UID: \"b5da0459-0be2-493c-a1ae-57e640e5b336\") "
Oct 30 17:07:53.213057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.212914    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-plugins-config-volume" pod "b5da0459-0be2-493c-a1ae-57e640e5b336" (UID: "b5da0459-0be2-493c-a1ae-57e640e5b336"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:07:53.213169 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.213072    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-routing-overlay-volume" pod "b5da0459-0be2-493c-a1ae-57e640e5b336" (UID: "b5da0459-0be2-493c-a1ae-57e640e5b336"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:07:53.213425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.213398    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:53.213425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.213424    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b5da0459-0be2-493c-a1ae-57e640e5b336-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:53.215016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.214995    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-kube-api-access-4qjzd" pod "b5da0459-0be2-493c-a1ae-57e640e5b336" (UID: "b5da0459-0be2-493c-a1ae-57e640e5b336"). InnerVolumeSpecName "kube-api-access-4qjzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:53.215105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.215061    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-provider-credentials" pod "b5da0459-0be2-493c-a1ae-57e640e5b336" (UID: "b5da0459-0be2-493c-a1ae-57e640e5b336"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:07:53.315286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.315243    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:53.315286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.315279    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qjzd\" (UniqueName: \"kubernetes.io/projected/b5da0459-0be2-493c-a1ae-57e640e5b336-kube-api-access-4qjzd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:07:53.732016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.731932    4497 generic.go:356] "Generic (PLEG): container finished" podID="b5da0459-0be2-493c-a1ae-57e640e5b336" containerID="7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3" exitCode=0
Oct 30 17:07:53.732490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.732058    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"
Oct 30 17:07:53.732490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.732059    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p" event={"ID":"b5da0459-0be2-493c-a1ae-57e640e5b336","Type":"ContainerDied","Data":"7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3"}
Oct 30 17:07:53.732490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.732175    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p" event={"ID":"b5da0459-0be2-493c-a1ae-57e640e5b336","Type":"ContainerDied","Data":"02769bebd66c3294ce197ff088cfb1b1451b48fd4a6730d9cea6b89377b85d85"}
Oct 30 17:07:53.732490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.732219    4497 scope.go:122] "RemoveContainer" containerID="7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3"
Oct 30 17:07:53.743004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.742984    4497 scope.go:122] "RemoveContainer" containerID="7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3"
Oct 30 17:07:53.743301 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:07:53.743283    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3\": container with ID starting with 7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3 not found: ID does not exist" containerID="7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3"
Oct 30 17:07:53.743358 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.743311    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3"} err="failed to get container status \"7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3\": rpc error: code = NotFound desc = could not find container \"7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3\": container with ID starting with 7c2812a9d57e6c4e7732c3210d05340b0299a064848c103c02316630868d8cd3 not found: ID does not exist"
Oct 30 17:07:53.778039 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.778008    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"]
Oct 30 17:07:53.787557 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:53.787530    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-54987744p"]
Oct 30 17:07:54.301832 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:07:54.301803    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5da0459-0be2-493c-a1ae-57e640e5b336" path="/var/lib/kubelet/pods/b5da0459-0be2-493c-a1ae-57e640e5b336/volumes"
Oct 30 17:08:03.735825 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:03.735794    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:08:03.927213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:03.927161    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"]
Oct 30 17:08:08.927514 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:08.927456    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6" podUID="ad55803d-19c6-4156-905e-ab89f0635dc2" containerName="payload-processing" containerID="cri-o://c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb" gracePeriod=25
Oct 30 17:08:09.199493 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.199425    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:08:09.244532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.244502    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-routing-overlay-volume\") pod \"ad55803d-19c6-4156-905e-ab89f0635dc2\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") "
Oct 30 17:08:09.244706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.244548    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-kube-api-access-mgtmb\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-kube-api-access-mgtmb\") pod \"ad55803d-19c6-4156-905e-ab89f0635dc2\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") "
Oct 30 17:08:09.244706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.244679    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-provider-credentials\") pod \"ad55803d-19c6-4156-905e-ab89f0635dc2\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") "
Oct 30 17:08:09.244813 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.244805    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-plugins-config-volume\") pod \"ad55803d-19c6-4156-905e-ab89f0635dc2\" (UID: \"ad55803d-19c6-4156-905e-ab89f0635dc2\") "
Oct 30 17:08:09.245002 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.244957    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-routing-overlay-volume" pod "ad55803d-19c6-4156-905e-ab89f0635dc2" (UID: "ad55803d-19c6-4156-905e-ab89f0635dc2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:08:09.245305 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.245279    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-plugins-config-volume" pod "ad55803d-19c6-4156-905e-ab89f0635dc2" (UID: "ad55803d-19c6-4156-905e-ab89f0635dc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:08:09.245455 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.245437    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:08:09.245502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.245457    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad55803d-19c6-4156-905e-ab89f0635dc2-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:08:09.247020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.246991    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-kube-api-access-mgtmb" pod "ad55803d-19c6-4156-905e-ab89f0635dc2" (UID: "ad55803d-19c6-4156-905e-ab89f0635dc2"). InnerVolumeSpecName "kube-api-access-mgtmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:08:09.247157 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.247136    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-provider-credentials" pod "ad55803d-19c6-4156-905e-ab89f0635dc2" (UID: "ad55803d-19c6-4156-905e-ab89f0635dc2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:08:09.346723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.346683    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:08:09.346723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.346721    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgtmb\" (UniqueName: \"kubernetes.io/projected/ad55803d-19c6-4156-905e-ab89f0635dc2-kube-api-access-mgtmb\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:08:09.844953 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.844920    4497 generic.go:356] "Generic (PLEG): container finished" podID="ad55803d-19c6-4156-905e-ab89f0635dc2" containerID="c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb" exitCode=0
Oct 30 17:08:09.845157 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.844986    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"
Oct 30 17:08:09.845157 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.845010    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6" event={"ID":"ad55803d-19c6-4156-905e-ab89f0635dc2","Type":"ContainerDied","Data":"c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb"}
Oct 30 17:08:09.845157 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.845091    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6" event={"ID":"ad55803d-19c6-4156-905e-ab89f0635dc2","Type":"ContainerDied","Data":"cc0d0ec8c8f92c71db5f963eec54204d621eff475ac71628707a34f5cef5f360"}
Oct 30 17:08:09.845157 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.845115    4497 scope.go:122] "RemoveContainer" containerID="c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb"
Oct 30 17:08:09.855721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.855699    4497 scope.go:122] "RemoveContainer" containerID="c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb"
Oct 30 17:08:09.856028 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:08:09.856007    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb\": container with ID starting with c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb not found: ID does not exist" containerID="c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb"
Oct 30 17:08:09.856144 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.856038    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb"} err="failed to get container status \"c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb\": rpc error: code = NotFound desc = could not find container \"c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb\": container with ID starting with c950b83d2f2c20651342862665da604dac9dd3f0b54bb8ae97e9b7eccaaa07cb not found: ID does not exist"
Oct 30 17:08:09.889816 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.889782    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"]
Oct 30 17:08:09.899152 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:09.899112    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667c9bgs6"]
Oct 30 17:08:10.301867 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:10.301833    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad55803d-19c6-4156-905e-ab89f0635dc2" path="/var/lib/kubelet/pods/ad55803d-19c6-4156-905e-ab89f0635dc2/volumes"
Oct 30 17:08:21.918078 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:08:21.918026    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:08:34.442759 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.442678    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"]
Oct 30 17:08:34.443507 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.443488    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b5da0459-0be2-493c-a1ae-57e640e5b336" containerName="payload-processing"
Oct 30 17:08:34.443507 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.443508    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5da0459-0be2-493c-a1ae-57e640e5b336" containerName="payload-processing"
Oct 30 17:08:34.443618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.443521    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ad55803d-19c6-4156-905e-ab89f0635dc2" containerName="payload-processing"
Oct 30 17:08:34.443618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.443526    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad55803d-19c6-4156-905e-ab89f0635dc2" containerName="payload-processing"
Oct 30 17:08:34.443683 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.443639    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5da0459-0be2-493c-a1ae-57e640e5b336" containerName="payload-processing"
Oct 30 17:08:34.443683 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.443651    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad55803d-19c6-4156-905e-ab89f0635dc2" containerName="payload-processing"
Oct 30 17:08:34.447395 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.447379    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.471992 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.471962    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"openshift-service-ca.crt\""
Oct 30 17:08:34.471992 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.471983    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f94fb5\""
Oct 30 17:08:34.472272 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.471963    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"agc-cred-2a77f2ea9da4\""
Oct 30 17:08:34.472272 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.471991    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"kube-root-ca.crt\""
Oct 30 17:08:34.472272 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.472039    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"payload-processing-external-model-e2e--ac654823-dockercfg-94ht7\""
Oct 30 17:08:34.472272 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.472201    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"routing-overlay\""
Oct 30 17:08:34.475744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.475721    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"]
Oct 30 17:08:34.505944 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.505837    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.506406 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.506380    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.506662 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.506644    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.506909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.506892    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzk4z\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-kube-api-access-dzk4z\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.610541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.608387    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.610541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.608452    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.610541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.608528    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk4z\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-kube-api-access-dzk4z\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.610541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.608656    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.610541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.609433    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.610541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.610095    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.618623 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.618528    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.632858 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.632814    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzk4z\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-kube-api-access-dzk4z\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:34.761301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:34.761269    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:35.072144 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:35.072122    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"]
Oct 30 17:08:35.075435 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:08:35.075401    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043085a9_e154_4d16_a1ea_5c12b452c61a.slice/crio-933ec075ba9db0f2bfc73d5f5c4b17ba84b8313c56767719aca7199755030a94 WatchSource:0}: Error finding container 933ec075ba9db0f2bfc73d5f5c4b17ba84b8313c56767719aca7199755030a94: Status 404 returned error can't find the container with id 933ec075ba9db0f2bfc73d5f5c4b17ba84b8313c56767719aca7199755030a94
Oct 30 17:08:36.047402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:36.047367    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj" event={"ID":"043085a9-e154-4d16-a1ea-5c12b452c61a","Type":"ContainerStarted","Data":"6bfbc1d1422a87890775b79e594cf133542962647c357720f274e35ed3410c99"}
Oct 30 17:08:36.047830 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:36.047446    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj" event={"ID":"043085a9-e154-4d16-a1ea-5c12b452c61a","Type":"ContainerStarted","Data":"933ec075ba9db0f2bfc73d5f5c4b17ba84b8313c56767719aca7199755030a94"}
Oct 30 17:08:36.047830 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:36.047509    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:36.087212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:36.087130    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj" podStartSLOduration=2.087115944 podStartE2EDuration="2.087115944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:36.084367116 +0000 UTC m=+2745.938245381" watchObservedRunningTime="2026-10-30 17:08:36.087115944 +0000 UTC m=+2745.940994208"
Oct 30 17:08:40.587054 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.587016    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"]
Oct 30 17:08:40.591523 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.591505    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.614382 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.614349    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"e2e-cfl-9d7fa9-provider-cred\""
Oct 30 17:08:40.614382 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.614350    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"routing-overlay\""
Oct 30 17:08:40.614577 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.614358    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2589b5\""
Oct 30 17:08:40.614577 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.614394    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"agc-cred-d59c88a1be40\""
Oct 30 17:08:40.616925 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.616907    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"]
Oct 30 17:08:40.617221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.617178    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-e2e--8e3f1132-dockercfg-gjbn7\""
Oct 30 17:08:40.617321 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.617232    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"kube-root-ca.crt\""
Oct 30 17:08:40.617321 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.617295    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"openshift-service-ca.crt\""
Oct 30 17:08:40.732625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.732585    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96w8j\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-kube-api-access-96w8j\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.732818 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.732651    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.732870 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.732829    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.732909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.732880    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.834078 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.834035    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.834286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.834177    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.834286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.834257    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.834441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.834424    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96w8j\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-kube-api-access-96w8j\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.834899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.834872    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.834996 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.834971    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.836684 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.836653    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.855741 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.855676    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96w8j\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-kube-api-access-96w8j\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:40.906765 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:40.906733    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:41.255031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:41.255005    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"]
Oct 30 17:08:41.258219 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:08:41.258161    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e0821c_59b9_49f5_b8b5_27bef3433dc3.slice/crio-f284bc8c3ebf45a5300e94ef6298d17caf78bbd70fdb7a19df999418d6d8afca WatchSource:0}: Error finding container f284bc8c3ebf45a5300e94ef6298d17caf78bbd70fdb7a19df999418d6d8afca: Status 404 returned error can't find the container with id f284bc8c3ebf45a5300e94ef6298d17caf78bbd70fdb7a19df999418d6d8afca
Oct 30 17:08:42.089922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:42.089869    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5" event={"ID":"70e0821c-59b9-49f5-b8b5-27bef3433dc3","Type":"ContainerStarted","Data":"96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71"}
Oct 30 17:08:42.090440 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:42.089940    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5" event={"ID":"70e0821c-59b9-49f5-b8b5-27bef3433dc3","Type":"ContainerStarted","Data":"f284bc8c3ebf45a5300e94ef6298d17caf78bbd70fdb7a19df999418d6d8afca"}
Oct 30 17:08:42.090440 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:42.089967    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:08:42.132091 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:42.132037    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5" podStartSLOduration=2.132022309 podStartE2EDuration="2.132022309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:42.129349897 +0000 UTC m=+2751.983228174" watchObservedRunningTime="2026-10-30 17:08:42.132022309 +0000 UTC m=+2751.985900568"
Oct 30 17:08:47.057291 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:47.057256    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:08:51.925105 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:08:51.925021    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:08:53.099583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:08:53.099555    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:09:21.918716 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:09:21.918675    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:09:51.917507 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:09:51.917458    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:09:55.965013 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:09:55.964982    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"]
Oct 30 17:10:00.966390 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:00.966310    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5" podUID="70e0821c-59b9-49f5-b8b5-27bef3433dc3" containerName="payload-processing" containerID="cri-o://96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71" gracePeriod=25
Oct 30 17:10:01.382435 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.382408    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:10:01.531387 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.531351    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-provider-credentials\") pod \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") "
Oct 30 17:10:01.531607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.531464    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-kube-api-access-96w8j\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-kube-api-access-96w8j\") pod \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") "
Oct 30 17:10:01.531607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.531586    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-plugins-config-volume\") pod \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") "
Oct 30 17:10:01.531756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.531731    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-routing-overlay-volume\") pod \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\" (UID: \"70e0821c-59b9-49f5-b8b5-27bef3433dc3\") "
Oct 30 17:10:01.532033 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.532007    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-plugins-config-volume" pod "70e0821c-59b9-49f5-b8b5-27bef3433dc3" (UID: "70e0821c-59b9-49f5-b8b5-27bef3433dc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:01.532126 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.532104    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-routing-overlay-volume" pod "70e0821c-59b9-49f5-b8b5-27bef3433dc3" (UID: "70e0821c-59b9-49f5-b8b5-27bef3433dc3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:01.532425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.532404    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:01.532479 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.532432    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0821c-59b9-49f5-b8b5-27bef3433dc3-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:01.533825 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.533797    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-provider-credentials" pod "70e0821c-59b9-49f5-b8b5-27bef3433dc3" (UID: "70e0821c-59b9-49f5-b8b5-27bef3433dc3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:01.533911 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.533871    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-kube-api-access-96w8j" pod "70e0821c-59b9-49f5-b8b5-27bef3433dc3" (UID: "70e0821c-59b9-49f5-b8b5-27bef3433dc3"). InnerVolumeSpecName "kube-api-access-96w8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:01.634500 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.634464    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96w8j\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-kube-api-access-96w8j\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:01.634500 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.634495    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0821c-59b9-49f5-b8b5-27bef3433dc3-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:01.726000 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.725967    4497 generic.go:356] "Generic (PLEG): container finished" podID="70e0821c-59b9-49f5-b8b5-27bef3433dc3" containerID="96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71" exitCode=0
Oct 30 17:10:01.726252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.726020    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"
Oct 30 17:10:01.726252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.726050    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5" event={"ID":"70e0821c-59b9-49f5-b8b5-27bef3433dc3","Type":"ContainerDied","Data":"96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71"}
Oct 30 17:10:01.726252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.726116    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5" event={"ID":"70e0821c-59b9-49f5-b8b5-27bef3433dc3","Type":"ContainerDied","Data":"f284bc8c3ebf45a5300e94ef6298d17caf78bbd70fdb7a19df999418d6d8afca"}
Oct 30 17:10:01.726252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.726149    4497 scope.go:122] "RemoveContainer" containerID="96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71"
Oct 30 17:10:01.742596 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.742564    4497 scope.go:122] "RemoveContainer" containerID="96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71"
Oct 30 17:10:01.747407 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:01.743094    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71\": container with ID starting with 96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71 not found: ID does not exist" containerID="96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71"
Oct 30 17:10:01.747407 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.743133    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71"} err="failed to get container status \"96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71\": rpc error: code = NotFound desc = could not find container \"96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71\": container with ID starting with 96e2da7a610d5c4b1feb5a38c1f10507879c7f7a4d7129a67ba1aa36c4eb8b71 not found: ID does not exist"
Oct 30 17:10:01.781614 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.781526    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"]
Oct 30 17:10:01.785326 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:01.785300    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-588fpnbs5"]
Oct 30 17:10:02.305239 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:02.305198    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e0821c-59b9-49f5-b8b5-27bef3433dc3" path="/var/lib/kubelet/pods/70e0821c-59b9-49f5-b8b5-27bef3433dc3/volumes"
Oct 30 17:10:14.681653 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.681593    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"]
Oct 30 17:10:14.682625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.682602    4497 cpu_manager.go:414] "containerMap: removing container" podUID="70e0821c-59b9-49f5-b8b5-27bef3433dc3" containerName="payload-processing"
Oct 30 17:10:14.682744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.682626    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e0821c-59b9-49f5-b8b5-27bef3433dc3" containerName="payload-processing"
Oct 30 17:10:14.682833 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.682819    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e0821c-59b9-49f5-b8b5-27bef3433dc3" containerName="payload-processing"
Oct 30 17:10:14.687792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.687768    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.703141 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.703115    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 17:10:14.712591 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.712561    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f590f7630637\""
Oct 30 17:10:14.712831 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.712804    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 17:10:14.712978 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.712856    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3b3debd510cb\""
Oct 30 17:10:14.712978 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.712893    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pmvgh\""
Oct 30 17:10:14.712978 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.712905    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 17:10:14.713148 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.713011    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 17:10:14.718154 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.718123    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"]
Oct 30 17:10:14.733034 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.733004    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-routing-overlay-volume\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.733282 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.733262    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-plugins-config-volume\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.733344 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.733300    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.733424 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.733410    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc4tg\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-kube-api-access-sc4tg\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.834884 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.834846    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-plugins-config-volume\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.834884 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.834890    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.835134 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.834952    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc4tg\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-kube-api-access-sc4tg\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.835134 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.835026    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-routing-overlay-volume\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.835134 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:14.835073    4497 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f590f7630637: secret "agc-cred-f590f7630637" not found
Oct 30 17:10:14.835134 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:14.835097    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75f7447496-tjcfz: secret "agc-cred-f590f7630637" not found
Oct 30 17:10:14.835356 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:14.835157    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials podName:0a1210dd-e934-4b00-bb9a-3290625bb3b9 nodeName:}" failed. No retries permitted until 2026-10-30 17:10:15.335141225 +0000 UTC m=+2845.189019468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials") pod "payload-processing-external-model-75f7447496-tjcfz" (UID: "0a1210dd-e934-4b00-bb9a-3290625bb3b9") : secret "agc-cred-f590f7630637" not found
Oct 30 17:10:14.835774 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.835751    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-plugins-config-volume\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.835883 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.835796    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-routing-overlay-volume\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:14.856624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:14.856594    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc4tg\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-kube-api-access-sc4tg\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:15.351515 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.351479    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:15.351698 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:15.351647    4497 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f590f7630637: secret "agc-cred-f590f7630637" not found
Oct 30 17:10:15.351698 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:15.351666    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75f7447496-tjcfz: secret "agc-cred-f590f7630637" not found
Oct 30 17:10:15.351783 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:15.351728    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials podName:0a1210dd-e934-4b00-bb9a-3290625bb3b9 nodeName:}" failed. No retries permitted until 2026-10-30 17:10:16.351714907 +0000 UTC m=+2846.205593163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials") pod "payload-processing-external-model-75f7447496-tjcfz" (UID: "0a1210dd-e934-4b00-bb9a-3290625bb3b9") : secret "agc-cred-f590f7630637" not found
Oct 30 17:10:15.521281 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.521245    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"]
Oct 30 17:10:15.521689 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:15.521662    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz" podUID="0a1210dd-e934-4b00-bb9a-3290625bb3b9"
Oct 30 17:10:15.823545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.823510    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:15.834260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.834228    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:15.961277 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.961232    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-kube-api-access-sc4tg\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-kube-api-access-sc4tg\") pod \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") "
Oct 30 17:10:15.961478 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.961305    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-plugins-config-volume\") pod \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") "
Oct 30 17:10:15.961558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.961528    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-routing-overlay-volume\") pod \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") "
Oct 30 17:10:15.961762 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.961737    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-plugins-config-volume" pod "0a1210dd-e934-4b00-bb9a-3290625bb3b9" (UID: "0a1210dd-e934-4b00-bb9a-3290625bb3b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:15.961866 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.961845    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-routing-overlay-volume" pod "0a1210dd-e934-4b00-bb9a-3290625bb3b9" (UID: "0a1210dd-e934-4b00-bb9a-3290625bb3b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:15.962041 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.962023    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:15.962102 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.962048    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1210dd-e934-4b00-bb9a-3290625bb3b9-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:15.963343 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:15.963326    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-kube-api-access-sc4tg" pod "0a1210dd-e934-4b00-bb9a-3290625bb3b9" (UID: "0a1210dd-e934-4b00-bb9a-3290625bb3b9"). InnerVolumeSpecName "kube-api-access-sc4tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:16.063171 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:16.063135    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc4tg\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-kube-api-access-sc4tg\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:16.366845 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:16.366811    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials\") pod \"payload-processing-external-model-75f7447496-tjcfz\" (UID: \"0a1210dd-e934-4b00-bb9a-3290625bb3b9\") " pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:16.367023 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:16.366952    4497 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f590f7630637: secret "agc-cred-f590f7630637" not found
Oct 30 17:10:16.367023 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:16.366969    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75f7447496-tjcfz: secret "agc-cred-f590f7630637" not found
Oct 30 17:10:16.367146 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:16.367034    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials podName:0a1210dd-e934-4b00-bb9a-3290625bb3b9 nodeName:}" failed. No retries permitted until 2026-10-30 17:10:18.367019682 +0000 UTC m=+2848.220897929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials") pod "payload-processing-external-model-75f7447496-tjcfz" (UID: "0a1210dd-e934-4b00-bb9a-3290625bb3b9") : secret "agc-cred-f590f7630637" not found
Oct 30 17:10:16.830213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:16.830169    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"
Oct 30 17:10:16.897753 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:16.897720    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"]
Oct 30 17:10:16.905254 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:16.905221    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75f7447496-tjcfz"]
Oct 30 17:10:16.975497 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:16.975466    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a1210dd-e934-4b00-bb9a-3290625bb3b9-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:18.301526 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.301492    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1210dd-e934-4b00-bb9a-3290625bb3b9" path="/var/lib/kubelet/pods/0a1210dd-e934-4b00-bb9a-3290625bb3b9/volumes"
Oct 30 17:10:18.326848 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.326809    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-565f564948-k4kj9"]
Oct 30 17:10:18.331376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.331351    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.349570 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.349539    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 17:10:18.363965 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.363934    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-565f564948-k4kj9"]
Oct 30 17:10:18.365100 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.364292    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 17:10:18.365282 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.364378    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 17:10:18.365394 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.364440    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 17:10:18.394254 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.394181    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-plugins-config-volume\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.394441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.394270    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-routing-overlay-volume\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.394441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.394317    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.394560 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.394538    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4477g\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-kube-api-access-4477g\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.410309 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.410279    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1f349073ea4c\""
Oct 30 17:10:18.410309 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.410300    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3b3debd510cb\""
Oct 30 17:10:18.410548 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.410285    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pmvgh\""
Oct 30 17:10:18.495963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.495919    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-plugins-config-volume\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.495963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.495966    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-routing-overlay-volume\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.496287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.496003    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.496287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.496122    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4477g\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-kube-api-access-4477g\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.496713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.496686    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-routing-overlay-volume\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.496847 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.496824    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-plugins-config-volume\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.498709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.498688    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.521891 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.521857    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4477g\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-kube-api-access-4477g\") pod \"payload-processing-external-model-565f564948-k4kj9\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") " pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.667286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.667167    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:18.976811 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:18.976788    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-565f564948-k4kj9"]
Oct 30 17:10:18.979945 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:18.979915    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f028c3_9024_4891_8b66_895e9f5e2397.slice/crio-77caa7734050c39284badbe5d24c7d9f6e18374c44b5aa296885f27435910218 WatchSource:0}: Error finding container 77caa7734050c39284badbe5d24c7d9f6e18374c44b5aa296885f27435910218: Status 404 returned error can't find the container with id 77caa7734050c39284badbe5d24c7d9f6e18374c44b5aa296885f27435910218
Oct 30 17:10:19.853393 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:19.853352    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9" event={"ID":"07f028c3-9024-4891-8b66-895e9f5e2397","Type":"ContainerStarted","Data":"ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08"}
Oct 30 17:10:19.853837 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:19.853412    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9" event={"ID":"07f028c3-9024-4891-8b66-895e9f5e2397","Type":"ContainerStarted","Data":"77caa7734050c39284badbe5d24c7d9f6e18374c44b5aa296885f27435910218"}
Oct 30 17:10:19.853837 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:19.853449    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:19.890652 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:19.890601    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9" podStartSLOduration=1.890580129 podStartE2EDuration="1.890580129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:19.889173021 +0000 UTC m=+2849.743051289" watchObservedRunningTime="2026-10-30 17:10:19.890580129 +0000 UTC m=+2849.744458393"
Oct 30 17:10:19.919331 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:19.919295    4497 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3b3debd510cb: secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:19.919331 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:19.919323    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-565f564948-k4kj9: secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:19.919628 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:19.919385    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials podName:07f028c3-9024-4891-8b66-895e9f5e2397 nodeName:}" failed. No retries permitted until 2026-10-30 17:10:20.41936528 +0000 UTC m=+2850.273243549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials") pod "payload-processing-external-model-565f564948-k4kj9" (UID: "07f028c3-9024-4891-8b66-895e9f5e2397") : secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:20.197611 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:20.197516    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-565f564948-k4kj9"]
Oct 30 17:10:20.439780 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:20.439744    4497 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3b3debd510cb: secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:20.439780 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:20.439773    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-565f564948-k4kj9: secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:20.440022 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:20.439837    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials podName:07f028c3-9024-4891-8b66-895e9f5e2397 nodeName:}" failed. No retries permitted until 2026-10-30 17:10:21.439816419 +0000 UTC m=+2851.293694662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials") pod "payload-processing-external-model-565f564948-k4kj9" (UID: "07f028c3-9024-4891-8b66-895e9f5e2397") : secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:20.724485 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:20.724448    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"]
Oct 30 17:10:21.468768 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:21.468722    4497 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3b3debd510cb: secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:21.468768 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:21.468762    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-565f564948-k4kj9: secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:21.469253 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:21.468857    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials podName:07f028c3-9024-4891-8b66-895e9f5e2397 nodeName:}" failed. No retries permitted until 2026-10-30 17:10:23.468839921 +0000 UTC m=+2853.322718177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials") pod "payload-processing-external-model-565f564948-k4kj9" (UID: "07f028c3-9024-4891-8b66-895e9f5e2397") : secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:21.918362 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:21.918326    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:10:23.498324 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:23.498289    4497 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1f349073ea4c: secret "agc-cred-1f349073ea4c" not found
Oct 30 17:10:23.498324 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:23.498315    4497 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3b3debd510cb: secret "agc-cred-3b3debd510cb" not found
Oct 30 17:10:23.498324 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:23.498326    4497 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-565f564948-k4kj9: [secret "agc-cred-1f349073ea4c" not found, secret "agc-cred-3b3debd510cb" not found]
Oct 30 17:10:23.498814 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:23.498379    4497 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials podName:07f028c3-9024-4891-8b66-895e9f5e2397 nodeName:}" failed. No retries permitted until 2026-10-30 17:10:27.498364878 +0000 UTC m=+2857.352243122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials") pod "payload-processing-external-model-565f564948-k4kj9" (UID: "07f028c3-9024-4891-8b66-895e9f5e2397") : [secret "agc-cred-1f349073ea4c" not found, secret "agc-cred-3b3debd510cb" not found]
Oct 30 17:10:25.725552 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:25.725510    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj" podUID="043085a9-e154-4d16-a1ea-5c12b452c61a" containerName="payload-processing" containerID="cri-o://6bfbc1d1422a87890775b79e594cf133542962647c357720f274e35ed3410c99" gracePeriod=25
Oct 30 17:10:25.896118 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:25.896087    4497 generic.go:356] "Generic (PLEG): container finished" podID="043085a9-e154-4d16-a1ea-5c12b452c61a" containerID="6bfbc1d1422a87890775b79e594cf133542962647c357720f274e35ed3410c99" exitCode=0
Oct 30 17:10:25.896303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:25.896145    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj" event={"ID":"043085a9-e154-4d16-a1ea-5c12b452c61a","Type":"ContainerDied","Data":"6bfbc1d1422a87890775b79e594cf133542962647c357720f274e35ed3410c99"}
Oct 30 17:10:26.006018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.005992    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:10:26.054949 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.054911    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-provider-credentials\") pod \"043085a9-e154-4d16-a1ea-5c12b452c61a\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") "
Oct 30 17:10:26.055155 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.055007    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-plugins-config-volume\") pod \"043085a9-e154-4d16-a1ea-5c12b452c61a\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") "
Oct 30 17:10:26.055287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.055231    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-routing-overlay-volume\") pod \"043085a9-e154-4d16-a1ea-5c12b452c61a\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") "
Oct 30 17:10:26.055371 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.055353    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-kube-api-access-dzk4z\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-kube-api-access-dzk4z\") pod \"043085a9-e154-4d16-a1ea-5c12b452c61a\" (UID: \"043085a9-e154-4d16-a1ea-5c12b452c61a\") "
Oct 30 17:10:26.055549 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.055497    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-plugins-config-volume" pod "043085a9-e154-4d16-a1ea-5c12b452c61a" (UID: "043085a9-e154-4d16-a1ea-5c12b452c61a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:26.055842 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.055821    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-routing-overlay-volume" pod "043085a9-e154-4d16-a1ea-5c12b452c61a" (UID: "043085a9-e154-4d16-a1ea-5c12b452c61a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:26.056053 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.056034    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:26.056163 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.056142    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/043085a9-e154-4d16-a1ea-5c12b452c61a-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:26.058036 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.058009    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-provider-credentials" pod "043085a9-e154-4d16-a1ea-5c12b452c61a" (UID: "043085a9-e154-4d16-a1ea-5c12b452c61a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:26.062230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.058422    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-kube-api-access-dzk4z" pod "043085a9-e154-4d16-a1ea-5c12b452c61a" (UID: "043085a9-e154-4d16-a1ea-5c12b452c61a"). InnerVolumeSpecName "kube-api-access-dzk4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:26.157611 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.157560    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzk4z\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-kube-api-access-dzk4z\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:26.157611 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.157601    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/043085a9-e154-4d16-a1ea-5c12b452c61a-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:26.866575 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.866505    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9" podUID="07f028c3-9024-4891-8b66-895e9f5e2397" containerName="payload-processing" containerID="cri-o://ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08" gracePeriod=25
Oct 30 17:10:26.904039 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.904015    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"
Oct 30 17:10:26.904171 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.904016    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj" event={"ID":"043085a9-e154-4d16-a1ea-5c12b452c61a","Type":"ContainerDied","Data":"933ec075ba9db0f2bfc73d5f5c4b17ba84b8313c56767719aca7199755030a94"}
Oct 30 17:10:26.904171 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.904141    4497 scope.go:122] "RemoveContainer" containerID="6bfbc1d1422a87890775b79e594cf133542962647c357720f274e35ed3410c99"
Oct 30 17:10:26.978889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.978859    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"]
Oct 30 17:10:26.987303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:26.987282    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-58d655hwj"]
Oct 30 17:10:27.144448 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.144385    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:27.178976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.178940    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-plugins-config-volume\") pod \"07f028c3-9024-4891-8b66-895e9f5e2397\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") "
Oct 30 17:10:27.179223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.179067    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-kube-api-access-4477g\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-kube-api-access-4477g\") pod \"07f028c3-9024-4891-8b66-895e9f5e2397\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") "
Oct 30 17:10:27.179223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.179171    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials\") pod \"07f028c3-9024-4891-8b66-895e9f5e2397\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") "
Oct 30 17:10:27.179363 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.179280    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-routing-overlay-volume\") pod \"07f028c3-9024-4891-8b66-895e9f5e2397\" (UID: \"07f028c3-9024-4891-8b66-895e9f5e2397\") "
Oct 30 17:10:27.180260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.180212    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-routing-overlay-volume" pod "07f028c3-9024-4891-8b66-895e9f5e2397" (UID: "07f028c3-9024-4891-8b66-895e9f5e2397"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:27.180459 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.180258    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-plugins-config-volume" pod "07f028c3-9024-4891-8b66-895e9f5e2397" (UID: "07f028c3-9024-4891-8b66-895e9f5e2397"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:27.184740 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.184708    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials" pod "07f028c3-9024-4891-8b66-895e9f5e2397" (UID: "07f028c3-9024-4891-8b66-895e9f5e2397"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:27.188454 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.188424    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-kube-api-access-4477g" pod "07f028c3-9024-4891-8b66-895e9f5e2397" (UID: "07f028c3-9024-4891-8b66-895e9f5e2397"). InnerVolumeSpecName "kube-api-access-4477g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:27.281756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.281723    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:27.281756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.281753    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4477g\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-kube-api-access-4477g\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:27.281756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.281763    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07f028c3-9024-4891-8b66-895e9f5e2397-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:27.282009 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.281772    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07f028c3-9024-4891-8b66-895e9f5e2397-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:27.670432 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.670396    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"]
Oct 30 17:10:27.671418 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.671402    4497 cpu_manager.go:414] "containerMap: removing container" podUID="043085a9-e154-4d16-a1ea-5c12b452c61a" containerName="payload-processing"
Oct 30 17:10:27.671477 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.671421    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043085a9-e154-4d16-a1ea-5c12b452c61a" containerName="payload-processing"
Oct 30 17:10:27.671477 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.671452    4497 cpu_manager.go:414] "containerMap: removing container" podUID="07f028c3-9024-4891-8b66-895e9f5e2397" containerName="payload-processing"
Oct 30 17:10:27.671477 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.671460    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f028c3-9024-4891-8b66-895e9f5e2397" containerName="payload-processing"
Oct 30 17:10:27.671617 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.671605    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="043085a9-e154-4d16-a1ea-5c12b452c61a" containerName="payload-processing"
Oct 30 17:10:27.671701 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.671622    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f028c3-9024-4891-8b66-895e9f5e2397" containerName="payload-processing"
Oct 30 17:10:27.675820 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.675799    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.694228 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.694197    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"]
Oct 30 17:10:27.699236 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.699177    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"agc-cred-8dda90d8fcab\""
Oct 30 17:10:27.699420 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.699236    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"kube-root-ca.crt\""
Oct 30 17:10:27.699420 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.699240    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"routing-overlay\""
Oct 30 17:10:27.699576 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.699298    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-e2e--02a9e3c4-dockercfg-cqwmp\""
Oct 30 17:10:27.699576 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.699305    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"openshift-service-ca.crt\""
Oct 30 17:10:27.699576 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.699357    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d5b798\""
Oct 30 17:10:27.788374 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.788338    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.788558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.788407    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frfr7\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-kube-api-access-frfr7\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.788622 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.788599    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.788740 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.788718    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.895757 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.895718    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.896481 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.896455    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.897110 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.897087    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.897383 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.897179    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frfr7\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-kube-api-access-frfr7\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.897574 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.896889    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.898511 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.898472    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.900875 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.900852    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.920323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.919940    4497 generic.go:356] "Generic (PLEG): container finished" podID="07f028c3-9024-4891-8b66-895e9f5e2397" containerID="ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08" exitCode=0
Oct 30 17:10:27.920323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.920038    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9" event={"ID":"07f028c3-9024-4891-8b66-895e9f5e2397","Type":"ContainerDied","Data":"ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08"}
Oct 30 17:10:27.920323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.920091    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9" event={"ID":"07f028c3-9024-4891-8b66-895e9f5e2397","Type":"ContainerDied","Data":"77caa7734050c39284badbe5d24c7d9f6e18374c44b5aa296885f27435910218"}
Oct 30 17:10:27.920323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.920122    4497 scope.go:122] "RemoveContainer" containerID="ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08"
Oct 30 17:10:27.920323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.920260    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-565f564948-k4kj9"
Oct 30 17:10:27.922760 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.922642    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frfr7\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-kube-api-access-frfr7\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:27.945827 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.945805    4497 scope.go:122] "RemoveContainer" containerID="ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08"
Oct 30 17:10:27.946153 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:27.946126    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08\": container with ID starting with ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08 not found: ID does not exist" containerID="ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08"
Oct 30 17:10:27.946312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.946160    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08"} err="failed to get container status \"ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08\": rpc error: code = NotFound desc = could not find container \"ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08\": container with ID starting with ad9811afd5911107f625e0daacd8bf2bad1058bc928a5f81daadee23d97abb08 not found: ID does not exist"
Oct 30 17:10:27.980085 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.980052    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-565f564948-k4kj9"]
Oct 30 17:10:27.984651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.984622    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-565f564948-k4kj9"]
Oct 30 17:10:27.992237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:27.992181    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:28.310338 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:28.310305    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043085a9-e154-4d16-a1ea-5c12b452c61a" path="/var/lib/kubelet/pods/043085a9-e154-4d16-a1ea-5c12b452c61a/volumes"
Oct 30 17:10:28.310849 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:28.310830    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f028c3-9024-4891-8b66-895e9f5e2397" path="/var/lib/kubelet/pods/07f028c3-9024-4891-8b66-895e9f5e2397/volumes"
Oct 30 17:10:28.311597 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:28.311571    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"]
Oct 30 17:10:28.312813 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:28.312791    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2450643_e0d3_4ccb_8550_fae05b0f38ea.slice/crio-9c4d69b6492b245f8fd49b947f7fe419b496932c5eea3681d9f4f97116eca5c7 WatchSource:0}: Error finding container 9c4d69b6492b245f8fd49b947f7fe419b496932c5eea3681d9f4f97116eca5c7: Status 404 returned error can't find the container with id 9c4d69b6492b245f8fd49b947f7fe419b496932c5eea3681d9f4f97116eca5c7
Oct 30 17:10:28.356731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:28.356700    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"]
Oct 30 17:10:28.935425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:28.935390    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk" event={"ID":"a2450643-e0d3-4ccb-8550-fae05b0f38ea","Type":"ContainerStarted","Data":"cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b"}
Oct 30 17:10:28.935888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:28.935442    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk" event={"ID":"a2450643-e0d3-4ccb-8550-fae05b0f38ea","Type":"ContainerStarted","Data":"9c4d69b6492b245f8fd49b947f7fe419b496932c5eea3681d9f4f97116eca5c7"}
Oct 30 17:10:28.935888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:28.935484    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:28.974604 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:28.974543    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk" podStartSLOduration=1.974529491 podStartE2EDuration="1.974529491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:28.970981596 +0000 UTC m=+2858.824859860" watchObservedRunningTime="2026-10-30 17:10:28.974529491 +0000 UTC m=+2858.828407755"
Oct 30 17:10:30.710812 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.710778    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"]
Oct 30 17:10:30.714815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.714790    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.729722 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.729695    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1f349073ea4c\""
Oct 30 17:10:30.739237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.739212    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rt99g\""
Oct 30 17:10:30.739447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.739348    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 17:10:30.739447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.739359    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 17:10:30.739447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.739396    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 17:10:30.739447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.739438    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 17:10:30.743752 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.743728    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"]
Oct 30 17:10:30.874210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.871445    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-provider-credentials\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.875648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.875620    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-routing-overlay-volume\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.889217 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.875996    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xz5c\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-kube-api-access-2xz5c\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.889552 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.889521    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-plugins-config-volume\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.991157 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.991057    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-provider-credentials\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.991157 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.991129    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-routing-overlay-volume\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.991460 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.991297    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xz5c\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-kube-api-access-2xz5c\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.991460 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.991327    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-plugins-config-volume\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.991947 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.991922    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-routing-overlay-volume\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.992073 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.992049    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-plugins-config-volume\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:30.993583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:30.993563    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-provider-credentials\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:31.012479 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:31.012450    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xz5c\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-kube-api-access-2xz5c\") pod \"payload-processing-external-model-75df6f9dc7-jkhpf\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:31.031665 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:31.031624    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:31.345077 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:31.345052    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"]
Oct 30 17:10:31.347629 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:31.347602    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f15f64_2f3b_4bef_b282_d2f6cd6a524d.slice/crio-17007cd46946680abac7d2863324eaa7f2ef4a44726454489597ab7228dc6d53 WatchSource:0}: Error finding container 17007cd46946680abac7d2863324eaa7f2ef4a44726454489597ab7228dc6d53: Status 404 returned error can't find the container with id 17007cd46946680abac7d2863324eaa7f2ef4a44726454489597ab7228dc6d53
Oct 30 17:10:31.963496 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:31.963455    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf" event={"ID":"26f15f64-2f3b-4bef-b282-d2f6cd6a524d","Type":"ContainerStarted","Data":"2f3abcb6d612c24ca0eb3ee98d894328d7abf1c2198eabb187f8c68b8cb76931"}
Oct 30 17:10:31.963917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:31.963516    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf" event={"ID":"26f15f64-2f3b-4bef-b282-d2f6cd6a524d","Type":"ContainerStarted","Data":"17007cd46946680abac7d2863324eaa7f2ef4a44726454489597ab7228dc6d53"}
Oct 30 17:10:31.963917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:31.963540    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:31.999563 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:31.999489    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf" podStartSLOduration=1.999471821 podStartE2EDuration="1.999471821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:31.999371991 +0000 UTC m=+2861.853250256" watchObservedRunningTime="2026-10-30 17:10:31.999471821 +0000 UTC m=+2861.853350087"
Oct 30 17:10:33.331407 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.331374    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"]
Oct 30 17:10:33.335840 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.335814    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.364360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.364326    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"]
Oct 30 17:10:33.427746 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.427702    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.427948 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.427757    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.427948 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.427914    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.428049 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.427972    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.428091 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.428062    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.428143 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.428117    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3603e008-6a64-4524-9f75-f28a32a8f0f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.428287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.428268    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5s5\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-kube-api-access-6r5s5\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.428374 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.428312    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.428475 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.428461    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.529731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.529694    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.529954 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.529748    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.529954 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.529860    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.529954 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.529918    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.529960    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530030    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530094    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3603e008-6a64-4524-9f75-f28a32a8f0f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530355 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530146    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530355 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530249    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5s5\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-kube-api-access-6r5s5\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530355 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530297    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530355 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530247    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530390    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530639 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530618    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.530861 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.530843    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3603e008-6a64-4524-9f75-f28a32a8f0f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.532231 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.532207    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.532491 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.532476    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.550958 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.550921    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.551082 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.551019    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5s5\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-kube-api-access-6r5s5\") pod \"maas-default-gateway-openshift-default-7854949c9-6q6mm\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.653688 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.653597    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:33.935927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.935829    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk" podUID="a2450643-e0d3-4ccb-8550-fae05b0f38ea" containerName="payload-processing" containerID="cri-o://cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b" gracePeriod=25
Oct 30 17:10:33.979923 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:33.979899    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"]
Oct 30 17:10:34.005371 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:34.005338    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3603e008_6a64_4524_9f75_f28a32a8f0f2.slice/crio-a14d2f53145280e6f32248fd41dea53c67796b9a8019a916ae12f0efc5972f38 WatchSource:0}: Error finding container a14d2f53145280e6f32248fd41dea53c67796b9a8019a916ae12f0efc5972f38: Status 404 returned error can't find the container with id a14d2f53145280e6f32248fd41dea53c67796b9a8019a916ae12f0efc5972f38
Oct 30 17:10:34.042234 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.042175    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:10:34.042368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.042286    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:10:34.042368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.042337    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:10:34.204701 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.204669    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:34.240655 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.240621    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-provider-credentials\") pod \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") "
Oct 30 17:10:34.240834 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.240818    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-kube-api-access-frfr7\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-kube-api-access-frfr7\") pod \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") "
Oct 30 17:10:34.240878 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.240862    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-routing-overlay-volume\") pod \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") "
Oct 30 17:10:34.240962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.240945    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-plugins-config-volume\") pod \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\" (UID: \"a2450643-e0d3-4ccb-8550-fae05b0f38ea\") "
Oct 30 17:10:34.241354 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.241322    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-routing-overlay-volume" pod "a2450643-e0d3-4ccb-8550-fae05b0f38ea" (UID: "a2450643-e0d3-4ccb-8550-fae05b0f38ea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:34.241497 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.241411    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-plugins-config-volume" pod "a2450643-e0d3-4ccb-8550-fae05b0f38ea" (UID: "a2450643-e0d3-4ccb-8550-fae05b0f38ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:34.241632 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.241614    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:34.241673 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.241639    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2450643-e0d3-4ccb-8550-fae05b0f38ea-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:34.242973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.242949    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-kube-api-access-frfr7" pod "a2450643-e0d3-4ccb-8550-fae05b0f38ea" (UID: "a2450643-e0d3-4ccb-8550-fae05b0f38ea"). InnerVolumeSpecName "kube-api-access-frfr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:34.243298 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.243276    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-provider-credentials" pod "a2450643-e0d3-4ccb-8550-fae05b0f38ea" (UID: "a2450643-e0d3-4ccb-8550-fae05b0f38ea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:34.343666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.343634    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frfr7\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-kube-api-access-frfr7\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:34.343666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:34.343661    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2450643-e0d3-4ccb-8550-fae05b0f38ea-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:35.006524 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.006481    4497 generic.go:356] "Generic (PLEG): container finished" podID="a2450643-e0d3-4ccb-8550-fae05b0f38ea" containerID="cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b" exitCode=0
Oct 30 17:10:35.006704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.006525    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk" event={"ID":"a2450643-e0d3-4ccb-8550-fae05b0f38ea","Type":"ContainerDied","Data":"cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b"}
Oct 30 17:10:35.006704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.006587    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk" event={"ID":"a2450643-e0d3-4ccb-8550-fae05b0f38ea","Type":"ContainerDied","Data":"9c4d69b6492b245f8fd49b947f7fe419b496932c5eea3681d9f4f97116eca5c7"}
Oct 30 17:10:35.006704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.006612    4497 scope.go:122] "RemoveContainer" containerID="cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b"
Oct 30 17:10:35.006704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.006536    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"
Oct 30 17:10:35.008710 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.008683    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" event={"ID":"3603e008-6a64-4524-9f75-f28a32a8f0f2","Type":"ContainerStarted","Data":"db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14"}
Oct 30 17:10:35.008802 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.008732    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" event={"ID":"3603e008-6a64-4524-9f75-f28a32a8f0f2","Type":"ContainerStarted","Data":"a14d2f53145280e6f32248fd41dea53c67796b9a8019a916ae12f0efc5972f38"}
Oct 30 17:10:35.017841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.017817    4497 scope.go:122] "RemoveContainer" containerID="cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b"
Oct 30 17:10:35.018113 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:35.018091    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b\": container with ID starting with cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b not found: ID does not exist" containerID="cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b"
Oct 30 17:10:35.018158 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.018123    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b"} err="failed to get container status \"cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b\": rpc error: code = NotFound desc = could not find container \"cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b\": container with ID starting with cf8f96a4f2a76328226d36fdb8b2ab9b7748184e6bcce12a81a0b046e634170b not found: ID does not exist"
Oct 30 17:10:35.052858 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.052824    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"]
Oct 30 17:10:35.061863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.061831    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-667cd5bpk"]
Oct 30 17:10:35.100310 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.100243    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podStartSLOduration=2.100222832 podStartE2EDuration="2.100222832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:35.096778151 +0000 UTC m=+2864.950656437" watchObservedRunningTime="2026-10-30 17:10:35.100222832 +0000 UTC m=+2864.954101099"
Oct 30 17:10:35.654819 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.654778    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:35.656367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.656342    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 17:10:35.656491 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.656387    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 17:10:35.928605 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:35.922696    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"]
Oct 30 17:10:36.350211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:36.349409    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2450643-e0d3-4ccb-8550-fae05b0f38ea" path="/var/lib/kubelet/pods/a2450643-e0d3-4ccb-8550-fae05b0f38ea/volumes"
Oct 30 17:10:36.654368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:36.654282    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 17:10:36.654368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:36.654347    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 17:10:37.654536 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:37.654502    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 17:10:37.654956 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:37.654581    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 17:10:38.654655 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:38.654618    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 17:10:38.655076 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:38.654677    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 17:10:39.665208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:39.664249    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 17:10:39.665208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:39.664311    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 17:10:40.654147 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:40.654108    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 17:10:40.654352 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:40.654179    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 17:10:40.923812 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:40.923713    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf" podUID="26f15f64-2f3b-4bef-b282-d2f6cd6a524d" containerName="payload-processing" containerID="cri-o://2f3abcb6d612c24ca0eb3ee98d894328d7abf1c2198eabb187f8c68b8cb76931" gracePeriod=25
Oct 30 17:10:40.973465 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:40.973430    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"]
Oct 30 17:10:40.974148 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:40.974130    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a2450643-e0d3-4ccb-8550-fae05b0f38ea" containerName="payload-processing"
Oct 30 17:10:40.974148 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:40.974146    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2450643-e0d3-4ccb-8550-fae05b0f38ea" containerName="payload-processing"
Oct 30 17:10:40.974339 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:40.974287    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2450643-e0d3-4ccb-8550-fae05b0f38ea" containerName="payload-processing"
Oct 30 17:10:40.977820 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:40.977791    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.000764 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.000730    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-c75cacd0b898\""
Oct 30 17:10:41.002173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.001325    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"kube-root-ca.crt\""
Oct 30 17:10:41.002173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.001413    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"routing-overlay\""
Oct 30 17:10:41.002173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.001472    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-e2e--ac3a616d-dockercfg-pg5xd\""
Oct 30 17:10:41.002173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.001535    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"openshift-service-ca.crt\""
Oct 30 17:10:41.002173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.001612    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3106d6\""
Oct 30 17:10:41.005120 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.005097    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"]
Oct 30 17:10:41.033998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.033311    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.033998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.033368    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.033998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.033632    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgmvs\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-kube-api-access-vgmvs\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.033998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.033726    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.073943 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.073907    4497 generic.go:356] "Generic (PLEG): container finished" podID="26f15f64-2f3b-4bef-b282-d2f6cd6a524d" containerID="2f3abcb6d612c24ca0eb3ee98d894328d7abf1c2198eabb187f8c68b8cb76931" exitCode=0
Oct 30 17:10:41.074116 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.073983    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf" event={"ID":"26f15f64-2f3b-4bef-b282-d2f6cd6a524d","Type":"ContainerDied","Data":"2f3abcb6d612c24ca0eb3ee98d894328d7abf1c2198eabb187f8c68b8cb76931"}
Oct 30 17:10:41.241262 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.238081    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.241262 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.238162    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.241262 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.238335    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgmvs\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-kube-api-access-vgmvs\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.241262 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.238390    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.241262 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.239776    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.241262 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.240658    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.278583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.265616    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.287578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.285977    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgmvs\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-kube-api-access-vgmvs\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.337929 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.337899    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:41.343751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.343723    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:41.445760 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.445708    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-plugins-config-volume\") pod \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") "
Oct 30 17:10:41.446062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.445878    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-kube-api-access-2xz5c\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-kube-api-access-2xz5c\") pod \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") "
Oct 30 17:10:41.446062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.446049    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-routing-overlay-volume\") pod \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") "
Oct 30 17:10:41.446328 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.446149    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-provider-credentials\") pod \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\" (UID: \"26f15f64-2f3b-4bef-b282-d2f6cd6a524d\") "
Oct 30 17:10:41.448430 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.447829    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-plugins-config-volume" pod "26f15f64-2f3b-4bef-b282-d2f6cd6a524d" (UID: "26f15f64-2f3b-4bef-b282-d2f6cd6a524d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:41.449613 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.449567    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-routing-overlay-volume" pod "26f15f64-2f3b-4bef-b282-d2f6cd6a524d" (UID: "26f15f64-2f3b-4bef-b282-d2f6cd6a524d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:41.459898 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.459841    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-provider-credentials" pod "26f15f64-2f3b-4bef-b282-d2f6cd6a524d" (UID: "26f15f64-2f3b-4bef-b282-d2f6cd6a524d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:41.467768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.467723    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-kube-api-access-2xz5c" pod "26f15f64-2f3b-4bef-b282-d2f6cd6a524d" (UID: "26f15f64-2f3b-4bef-b282-d2f6cd6a524d"). InnerVolumeSpecName "kube-api-access-2xz5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:41.548994 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.548953    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:41.548994 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.548996    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xz5c\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-kube-api-access-2xz5c\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:41.549237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.549016    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:41.549237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.549032    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26f15f64-2f3b-4bef-b282-d2f6cd6a524d-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:41.654153 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.654047    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"]
Oct 30 17:10:41.654153 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.654065    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 17:10:41.654153 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.654132    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 17:10:41.657359 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:41.657331    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5879e29e_6852_4eee_a56f_6a662d5f32ee.slice/crio-0bbf903e014948ebaf7263c4875e159d1efd3ea6fae1cf9a45a4a8b85a8c9b41 WatchSource:0}: Error finding container 0bbf903e014948ebaf7263c4875e159d1efd3ea6fae1cf9a45a4a8b85a8c9b41: Status 404 returned error can't find the container with id 0bbf903e014948ebaf7263c4875e159d1efd3ea6fae1cf9a45a4a8b85a8c9b41
Oct 30 17:10:41.949761 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.949670    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"]
Oct 30 17:10:41.950418 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.950401    4497 cpu_manager.go:414] "containerMap: removing container" podUID="26f15f64-2f3b-4bef-b282-d2f6cd6a524d" containerName="payload-processing"
Oct 30 17:10:41.950480 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.950419    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f15f64-2f3b-4bef-b282-d2f6cd6a524d" containerName="payload-processing"
Oct 30 17:10:41.950598 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.950588    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f15f64-2f3b-4bef-b282-d2f6cd6a524d" containerName="payload-processing"
Oct 30 17:10:41.954292 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.954270    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:41.968131 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.968099    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5758b7\""
Oct 30 17:10:41.968311 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.968166    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"openshift-service-ca.crt\""
Oct 30 17:10:41.968311 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.968168    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"kube-root-ca.crt\""
Oct 30 17:10:41.968311 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.968278    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"routing-overlay\""
Oct 30 17:10:41.968688 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.968669    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"agc-cred-0d17396f5b80\""
Oct 30 17:10:41.968799 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.968786    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"payload-processing-external-model-e2e--5c3004c8-dockercfg-nch74\""
Oct 30 17:10:41.976332 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:41.976310    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"]
Oct 30 17:10:42.066211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.065345    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.066211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.065431    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fng45\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-kube-api-access-fng45\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.066211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.065491    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.066211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.065588    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.152407 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.152367    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf" event={"ID":"26f15f64-2f3b-4bef-b282-d2f6cd6a524d","Type":"ContainerDied","Data":"17007cd46946680abac7d2863324eaa7f2ef4a44726454489597ab7228dc6d53"}
Oct 30 17:10:42.152605 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.152443    4497 scope.go:122] "RemoveContainer" containerID="2f3abcb6d612c24ca0eb3ee98d894328d7abf1c2198eabb187f8c68b8cb76931"
Oct 30 17:10:42.152683 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.152613    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"
Oct 30 17:10:42.158573 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.158535    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f" event={"ID":"5879e29e-6852-4eee-a56f-6a662d5f32ee","Type":"ContainerStarted","Data":"a18fedbfe45a2632713ad03498bc6a002f90000dbd33d496684d20acff8502ef"}
Oct 30 17:10:42.158737 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.158597    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f" event={"ID":"5879e29e-6852-4eee-a56f-6a662d5f32ee","Type":"ContainerStarted","Data":"0bbf903e014948ebaf7263c4875e159d1efd3ea6fae1cf9a45a4a8b85a8c9b41"}
Oct 30 17:10:42.158737 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.158633    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:42.168529 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.168457    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.168677 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.168549    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fng45\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-kube-api-access-fng45\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.168677 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.168611    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.168793 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.168732    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.169368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.169283    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.169981 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.169949    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.171970 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.171943    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.189941 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.189914    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fng45\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-kube-api-access-fng45\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.209105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.208998    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f" podStartSLOduration=2.208977522 podStartE2EDuration="2.208977522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:42.20397095 +0000 UTC m=+2872.057849216" watchObservedRunningTime="2026-10-30 17:10:42.208977522 +0000 UTC m=+2872.062855788"
Oct 30 17:10:42.237720 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.237684    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"]
Oct 30 17:10:42.241002 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.240974    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-jkhpf"]
Oct 30 17:10:42.274467 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.274429    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:42.303617 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.303581    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f15f64-2f3b-4bef-b282-d2f6cd6a524d" path="/var/lib/kubelet/pods/26f15f64-2f3b-4bef-b282-d2f6cd6a524d/volumes"
Oct 30 17:10:42.593479 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:42.593447    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"]
Oct 30 17:10:42.595868 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:42.595682    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1bed12_0da3_45cd_a2d6_01b81d3c480f.slice/crio-79746f3af8129778ab7bc4fe3fd95c19a8ed61307d5a2c61360c9788b185cf19 WatchSource:0}: Error finding container 79746f3af8129778ab7bc4fe3fd95c19a8ed61307d5a2c61360c9788b185cf19: Status 404 returned error can't find the container with id 79746f3af8129778ab7bc4fe3fd95c19a8ed61307d5a2c61360c9788b185cf19
Oct 30 17:10:43.170202 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:43.170139    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl" event={"ID":"2f1bed12-0da3-45cd-a2d6-01b81d3c480f","Type":"ContainerStarted","Data":"0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389"}
Oct 30 17:10:43.170688 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:43.170232    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:43.170688 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:43.170257    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl" event={"ID":"2f1bed12-0da3-45cd-a2d6-01b81d3c480f","Type":"ContainerStarted","Data":"79746f3af8129778ab7bc4fe3fd95c19a8ed61307d5a2c61360c9788b185cf19"}
Oct 30 17:10:43.209210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:43.208595    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl" podStartSLOduration=2.208574968 podStartE2EDuration="2.208574968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:43.205993379 +0000 UTC m=+2873.059871669" watchObservedRunningTime="2026-10-30 17:10:43.208574968 +0000 UTC m=+2873.062453257"
Oct 30 17:10:43.664167 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:43.664103    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:10:43.664386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:43.664181    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:10:44.705352 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:44.705302    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:10:44.705935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:44.705357    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:10:45.747365 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:45.747314    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:10:45.747851 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:45.747401    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:10:46.075228 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:46.075163    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:46.075787 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:46.075759    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:46.076578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:46.076557    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:10:48.795212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.795156    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"]
Oct 30 17:10:48.806306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.806275    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:48.828145 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.826070    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"]
Oct 30 17:10:48.837593 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.837564    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"openshift-service-ca.crt\""
Oct 30 17:10:48.837817 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.837564    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"agc-cred-fc53cb817a37\""
Oct 30 17:10:48.837938 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.837662    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"kube-root-ca.crt\""
Oct 30 17:10:48.838021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.837790    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"payload-processing-external-model-e2e--dc82f608-dockercfg-qzdc9\""
Oct 30 17:10:48.846987 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.846963    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"payload-processing-external-model-plugins-e2e-extmod-b-abda73\""
Oct 30 17:10:48.847117 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.847016    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"routing-overlay\""
Oct 30 17:10:48.971062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.971028    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncr5c\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-kube-api-access-ncr5c\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:48.971645 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.971612    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:48.971851 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.971833    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:48.972071 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:48.972054    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.073700 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.073609    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.073888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.073744    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncr5c\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-kube-api-access-ncr5c\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.073946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.073922    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.073994 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.073951    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.074602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.074565    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.074732 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.074673    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.076546 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.076527    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.094787 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.094756    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncr5c\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-kube-api-access-ncr5c\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.146146 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.146113    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:49.479588 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.479514    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"]
Oct 30 17:10:49.483376 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:49.483348    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece549bc_d032_487d_8c0b_466e49eae5fb.slice/crio-2fa9bd403f0ca81b28c560e011329dc543e739e1858ef7fa42f462f8ac642a2b WatchSource:0}: Error finding container 2fa9bd403f0ca81b28c560e011329dc543e739e1858ef7fa42f462f8ac642a2b: Status 404 returned error can't find the container with id 2fa9bd403f0ca81b28c560e011329dc543e739e1858ef7fa42f462f8ac642a2b
Oct 30 17:10:49.764235 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.757317    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"]
Oct 30 17:10:49.764235 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.762479    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:49.774641 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.774614    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"agc-cred-36a83f9d45dd\""
Oct 30 17:10:49.780269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.780239    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"]
Oct 30 17:10:49.783298 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.783246    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-e2e--02a9e3c4-dockercfg-25qz6\""
Oct 30 17:10:49.783780 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.783751    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d5b798\""
Oct 30 17:10:49.784004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.783805    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"routing-overlay\""
Oct 30 17:10:49.784004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.783831    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"kube-root-ca.crt\""
Oct 30 17:10:49.784154 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.784127    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"openshift-service-ca.crt\""
Oct 30 17:10:49.892124 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.892080    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqm24\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-kube-api-access-hqm24\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:49.892607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.892292    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:49.892607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.892361    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:49.892607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.892431    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:49.998371 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.998332    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqm24\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-kube-api-access-hqm24\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:49.998902 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:49.998880    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:50.000225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.000202    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:50.000416 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.000401    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:50.013207 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.000092    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:50.013207 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.012495    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:50.013784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.013746    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:50.033260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.032445    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqm24\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-kube-api-access-hqm24\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:50.092738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.091471    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:50.319848 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.318681    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8" event={"ID":"ece549bc-d032-487d-8c0b-466e49eae5fb","Type":"ContainerStarted","Data":"d5048ee3d1d4fb6db4329d04b0e83d9c5c5b12bdde5cc9817916de7a93fb9d4a"}
Oct 30 17:10:50.319848 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.318748    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:50.319848 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.318774    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8" event={"ID":"ece549bc-d032-487d-8c0b-466e49eae5fb","Type":"ContainerStarted","Data":"2fa9bd403f0ca81b28c560e011329dc543e739e1858ef7fa42f462f8ac642a2b"}
Oct 30 17:10:50.372826 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.372769    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8" podStartSLOduration=2.372749802 podStartE2EDuration="2.372749802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:50.372691525 +0000 UTC m=+2880.226569774" watchObservedRunningTime="2026-10-30 17:10:50.372749802 +0000 UTC m=+2880.226628069"
Oct 30 17:10:50.463078 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.463047    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"]
Oct 30 17:10:50.468817 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.468791    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.495254 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.487109    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-de47d9dd1e5e\""
Oct 30 17:10:50.503386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.503335    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"]
Oct 30 17:10:50.563172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.563070    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"]
Oct 30 17:10:50.568362 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:50.568321    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8a3e32_72bb_463b_8cc6_93999ce0d2a4.slice/crio-209579b8c28685f5e25f246d82779ec1b7cffe54144da7db588bfb50c3a72776 WatchSource:0}: Error finding container 209579b8c28685f5e25f246d82779ec1b7cffe54144da7db588bfb50c3a72776: Status 404 returned error can't find the container with id 209579b8c28685f5e25f246d82779ec1b7cffe54144da7db588bfb50c3a72776
Oct 30 17:10:50.617629 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.617595    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtb8\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-kube-api-access-fgtb8\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.617828 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.617801    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.617916 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.617876    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.618111 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.618080    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.720127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.720089    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.720347 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.720246    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.720429 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.720346    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtb8\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-kube-api-access-fgtb8\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.720578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.720553    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.721333 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.721306    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.722883 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.722862    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.722973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.722936    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.743608 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.743573    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtb8\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-kube-api-access-fgtb8\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:50.807274 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:50.807232    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:51.169234 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:51.169142    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"]
Oct 30 17:10:51.174060 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:51.173122    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c82c9a_6de8_4ec6_a119_9b1fa332612c.slice/crio-40889f492adacda7d7193514231d414f1844469b7b869aed9c750bffc572770f WatchSource:0}: Error finding container 40889f492adacda7d7193514231d414f1844469b7b869aed9c750bffc572770f: Status 404 returned error can't find the container with id 40889f492adacda7d7193514231d414f1844469b7b869aed9c750bffc572770f
Oct 30 17:10:51.330211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:51.326270    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r" event={"ID":"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4","Type":"ContainerStarted","Data":"e419731986b08f9e838c0b4df4ec28cd8d5635cab0c3d0d34b6d300b60a54df1"}
Oct 30 17:10:51.330211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:51.326342    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r" event={"ID":"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4","Type":"ContainerStarted","Data":"209579b8c28685f5e25f246d82779ec1b7cffe54144da7db588bfb50c3a72776"}
Oct 30 17:10:51.330211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:51.327509    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:10:51.334234 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:51.332305    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4" event={"ID":"08c82c9a-6de8-4ec6-a119-9b1fa332612c","Type":"ContainerStarted","Data":"69a2a11a79f60912368541781df99df4af4999d189d34d12824e156eb61c4399"}
Oct 30 17:10:51.334234 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:51.332356    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4" event={"ID":"08c82c9a-6de8-4ec6-a119-9b1fa332612c","Type":"ContainerStarted","Data":"40889f492adacda7d7193514231d414f1844469b7b869aed9c750bffc572770f"}
Oct 30 17:10:51.334234 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:51.332386    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:10:51.364070 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:51.364011    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r" podStartSLOduration=2.36399663 podStartE2EDuration="2.36399663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:51.363431544 +0000 UTC m=+2881.217309812" watchObservedRunningTime="2026-10-30 17:10:51.36399663 +0000 UTC m=+2881.217874895"
Oct 30 17:10:51.943255 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:10:51.943150    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:10:53.175707 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:53.175672    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:10:53.213140 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:53.213088    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4" podStartSLOduration=3.213074869 podStartE2EDuration="3.213074869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:51.4043504 +0000 UTC m=+2881.258228666" watchObservedRunningTime="2026-10-30 17:10:53.213074869 +0000 UTC m=+2883.066953133"
Oct 30 17:10:54.058455 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:54.058420    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"]
Oct 30 17:10:54.061761 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:54.061737    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:54.183032 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:54.182996    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:10:54.913641 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:54.913606    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"]
Oct 30 17:10:57.782700 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:57.782666    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"]
Oct 30 17:10:57.788919 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:57.788892    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:57.801218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:57.801169    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-06e7f6-openshift-default-dockercfg-pf8rn\""
Oct 30 17:10:57.815198 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:57.815150    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"]
Oct 30 17:10:58.055214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055113    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.055214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055181    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m29g\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-kube-api-access-5m29g\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.055450 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055336    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f07ab-72d9-414a-abd2-d379e71a331f-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.055513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055454    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.055557 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055511    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.055557 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055539    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.055709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055693    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.055759 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055730    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.055818 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.055767    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157426    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157470    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157507    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157533    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157573    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m29g\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-kube-api-access-5m29g\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157603    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f07ab-72d9-414a-abd2-d379e71a331f-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157661    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157695    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.157719    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.158104    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.158963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.158732    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.159095 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.159067    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.159876 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.159379    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.160005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.159822    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f07ab-72d9-414a-abd2-d379e71a331f-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.163381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.163353    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.168528 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.168489    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.180164 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.180124    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.180411 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.180302    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m29g\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-kube-api-access-5m29g\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.404490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.404404    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:58.736376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.736218    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"]
Oct 30 17:10:58.739203 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:10:58.739161    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1f07ab_72d9_414a_abd2_d379e71a331f.slice/crio-ae095aa4929f9e434576cc13796b7f32f06f717544508d789192473801cdf4ab WatchSource:0}: Error finding container ae095aa4929f9e434576cc13796b7f32f06f717544508d789192473801cdf4ab: Status 404 returned error can't find the container with id ae095aa4929f9e434576cc13796b7f32f06f717544508d789192473801cdf4ab
Oct 30 17:10:58.742103 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.742064    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:10:58.742216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.742138    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:10:58.742216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:58.742169    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:10:59.058827 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.058779    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8" podUID="ece549bc-d032-487d-8c0b-466e49eae5fb" containerName="payload-processing" containerID="cri-o://d5048ee3d1d4fb6db4329d04b0e83d9c5c5b12bdde5cc9817916de7a93fb9d4a" gracePeriod=25
Oct 30 17:10:59.402109 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.402080    4497 generic.go:356] "Generic (PLEG): container finished" podID="ece549bc-d032-487d-8c0b-466e49eae5fb" containerID="d5048ee3d1d4fb6db4329d04b0e83d9c5c5b12bdde5cc9817916de7a93fb9d4a" exitCode=0
Oct 30 17:10:59.402252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.402154    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8" event={"ID":"ece549bc-d032-487d-8c0b-466e49eae5fb","Type":"ContainerDied","Data":"d5048ee3d1d4fb6db4329d04b0e83d9c5c5b12bdde5cc9817916de7a93fb9d4a"}
Oct 30 17:10:59.403978 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.403947    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" event={"ID":"ec1f07ab-72d9-414a-abd2-d379e71a331f","Type":"ContainerStarted","Data":"772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68"}
Oct 30 17:10:59.404086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.403996    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" event={"ID":"ec1f07ab-72d9-414a-abd2-d379e71a331f","Type":"ContainerStarted","Data":"ae095aa4929f9e434576cc13796b7f32f06f717544508d789192473801cdf4ab"}
Oct 30 17:10:59.404749 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.404734    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:10:59.406577 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.406557    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:10:59.406676 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.406604    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:10:59.434309 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.434280    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:10:59.456097 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.456028    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podStartSLOduration=2.456006788 podStartE2EDuration="2.456006788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:59.454342791 +0000 UTC m=+2889.308221070" watchObservedRunningTime="2026-10-30 17:10:59.456006788 +0000 UTC m=+2889.309885055"
Oct 30 17:10:59.493514 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.493483    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-provider-credentials\") pod \"ece549bc-d032-487d-8c0b-466e49eae5fb\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") "
Oct 30 17:10:59.493703 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.493586    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-routing-overlay-volume\") pod \"ece549bc-d032-487d-8c0b-466e49eae5fb\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") "
Oct 30 17:10:59.493782 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.493741    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-plugins-config-volume\") pod \"ece549bc-d032-487d-8c0b-466e49eae5fb\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") "
Oct 30 17:10:59.493858 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.493780    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-kube-api-access-ncr5c\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-kube-api-access-ncr5c\") pod \"ece549bc-d032-487d-8c0b-466e49eae5fb\" (UID: \"ece549bc-d032-487d-8c0b-466e49eae5fb\") "
Oct 30 17:10:59.493994 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.493971    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-routing-overlay-volume" pod "ece549bc-d032-487d-8c0b-466e49eae5fb" (UID: "ece549bc-d032-487d-8c0b-466e49eae5fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:59.494211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.494158    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-plugins-config-volume" pod "ece549bc-d032-487d-8c0b-466e49eae5fb" (UID: "ece549bc-d032-487d-8c0b-466e49eae5fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:10:59.494888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.494852    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:59.495150 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.494880    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece549bc-d032-487d-8c0b-466e49eae5fb-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:59.497709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.496544    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-provider-credentials" pod "ece549bc-d032-487d-8c0b-466e49eae5fb" (UID: "ece549bc-d032-487d-8c0b-466e49eae5fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:59.498035 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.497998    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-kube-api-access-ncr5c" pod "ece549bc-d032-487d-8c0b-466e49eae5fb" (UID: "ece549bc-d032-487d-8c0b-466e49eae5fb"). InnerVolumeSpecName "kube-api-access-ncr5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:10:59.596773 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.596679    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncr5c\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-kube-api-access-ncr5c\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:59.596773 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.596716    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece549bc-d032-487d-8c0b-466e49eae5fb-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:10:59.914645 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.914310    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r" podUID="4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" containerName="payload-processing" containerID="cri-o://e419731986b08f9e838c0b4df4ec28cd8d5635cab0c3d0d34b6d300b60a54df1" gracePeriod=25
Oct 30 17:10:59.950470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.950430    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"]
Oct 30 17:10:59.951902 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.951870    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ece549bc-d032-487d-8c0b-466e49eae5fb" containerName="payload-processing"
Oct 30 17:10:59.953265 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.953242    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece549bc-d032-487d-8c0b-466e49eae5fb" containerName="payload-processing"
Oct 30 17:10:59.953703 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.953686    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece549bc-d032-487d-8c0b-466e49eae5fb" containerName="payload-processing"
Oct 30 17:10:59.962041 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.962011    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:10:59.963310 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:10:59.963207    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"]
Oct 30 17:11:00.006906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.004389    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bb5b1\"/\"routing-overlay\""
Oct 30 17:11:00.006906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.004351    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bb5b1\"/\"openshift-service-ca.crt\""
Oct 30 17:11:00.006906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.004834    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bb5b1\"/\"kube-root-ca.crt\""
Oct 30 17:11:00.006906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.005036    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9bb5b1\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9bb5b1\""
Oct 30 17:11:00.006906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.005344    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bb5b1\"/\"payload-processing-external-model-e2e--27dad712-dockercfg-5fc5z\""
Oct 30 17:11:00.006906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.005505    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9bb5b1\"/\"agc-cred-12a5ccb49783\""
Oct 30 17:11:00.018404 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.018359    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9zn7\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-kube-api-access-w9zn7\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.018622 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.018443    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.018622 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.018487    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.018782 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.018757    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.132637 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.132596    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9zn7\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-kube-api-access-w9zn7\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.150208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.149415    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.150208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.149508    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.150208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.150032    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.154266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.151365    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.154266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.153065    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.178273 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.172137    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9zn7\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-kube-api-access-w9zn7\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.213207 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.201859    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") " pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.348651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.348614    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:00.422348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.421790    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:11:00.422348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.421859    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:11:00.434639 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.434606    4497 generic.go:356] "Generic (PLEG): container finished" podID="4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" containerID="e419731986b08f9e838c0b4df4ec28cd8d5635cab0c3d0d34b6d300b60a54df1" exitCode=0
Oct 30 17:11:00.434871 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.434844    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r" event={"ID":"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4","Type":"ContainerDied","Data":"e419731986b08f9e838c0b4df4ec28cd8d5635cab0c3d0d34b6d300b60a54df1"}
Oct 30 17:11:00.437678 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.437651    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"
Oct 30 17:11:00.438479 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.438450    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8" event={"ID":"ece549bc-d032-487d-8c0b-466e49eae5fb","Type":"ContainerDied","Data":"2fa9bd403f0ca81b28c560e011329dc543e739e1858ef7fa42f462f8ac642a2b"}
Oct 30 17:11:00.438675 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.438659    4497 scope.go:122] "RemoveContainer" containerID="d5048ee3d1d4fb6db4329d04b0e83d9c5c5b12bdde5cc9817916de7a93fb9d4a"
Oct 30 17:11:00.466675 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.466647    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:11:00.512083 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.512048    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"]
Oct 30 17:11:00.516489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.516461    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cfcdkbv8"]
Oct 30 17:11:00.562764 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.562269    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-kube-api-access-hqm24\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-kube-api-access-hqm24\") pod \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") "
Oct 30 17:11:00.563017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.562999    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-routing-overlay-volume\") pod \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") "
Oct 30 17:11:00.563068 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.563028    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-provider-credentials\") pod \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") "
Oct 30 17:11:00.563108 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.563092    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-plugins-config-volume\") pod \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\" (UID: \"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4\") "
Oct 30 17:11:00.563952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.563915    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-plugins-config-volume" pod "4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" (UID: "4d8a3e32-72bb-463b-8cc6-93999ce0d2a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:11:00.564292 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.564268    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:00.564534 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.564506    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-routing-overlay-volume" pod "4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" (UID: "4d8a3e32-72bb-463b-8cc6-93999ce0d2a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:11:00.565147 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.565127    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-kube-api-access-hqm24" pod "4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" (UID: "4d8a3e32-72bb-463b-8cc6-93999ce0d2a4"). InnerVolumeSpecName "kube-api-access-hqm24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:11:00.566599 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.566577    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-provider-credentials" pod "4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" (UID: "4d8a3e32-72bb-463b-8cc6-93999ce0d2a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:11:00.665296 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.665254    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:00.665296 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.665286    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:00.665296 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.665298    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqm24\" (UniqueName: \"kubernetes.io/projected/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4-kube-api-access-hqm24\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:00.746570 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:00.746541    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"]
Oct 30 17:11:00.749642 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:11:00.749615    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64df251_52af_4819_ae5b_79db38f8fc0c.slice/crio-e1e6f2deb917e27e713feb57a4b74e87a46ec71b78e6895d0f9faf63473f1d2d WatchSource:0}: Error finding container e1e6f2deb917e27e713feb57a4b74e87a46ec71b78e6895d0f9faf63473f1d2d: Status 404 returned error can't find the container with id e1e6f2deb917e27e713feb57a4b74e87a46ec71b78e6895d0f9faf63473f1d2d
Oct 30 17:11:01.405685 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.405651    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:11:01.406103 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.405713    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:11:01.446724 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.446629    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2" event={"ID":"d64df251-52af-4819-ae5b-79db38f8fc0c","Type":"ContainerStarted","Data":"9f198b7d364d77ebbf7dfab9e1ba4b42a556b716d1043246ca82f425358860b7"}
Oct 30 17:11:01.446724 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.446681    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2" event={"ID":"d64df251-52af-4819-ae5b-79db38f8fc0c","Type":"ContainerStarted","Data":"e1e6f2deb917e27e713feb57a4b74e87a46ec71b78e6895d0f9faf63473f1d2d"}
Oct 30 17:11:01.446958 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.446732    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:01.448089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.448062    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"
Oct 30 17:11:01.448223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.448114    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r" event={"ID":"4d8a3e32-72bb-463b-8cc6-93999ce0d2a4","Type":"ContainerDied","Data":"209579b8c28685f5e25f246d82779ec1b7cffe54144da7db588bfb50c3a72776"}
Oct 30 17:11:01.448223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.448164    4497 scope.go:122] "RemoveContainer" containerID="e419731986b08f9e838c0b4df4ec28cd8d5635cab0c3d0d34b6d300b60a54df1"
Oct 30 17:11:01.500306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.500242    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2" podStartSLOduration=2.500226526 podStartE2EDuration="2.500226526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:01.496661803 +0000 UTC m=+2891.350540077" watchObservedRunningTime="2026-10-30 17:11:01.500226526 +0000 UTC m=+2891.354104790"
Oct 30 17:11:01.529270 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.529229    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"]
Oct 30 17:11:01.534542 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:01.534511    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-ccdd8fp8r"]
Oct 30 17:11:02.198272 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.197932    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"]
Oct 30 17:11:02.198908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.198886    4497 cpu_manager.go:414] "containerMap: removing container" podUID="4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" containerName="payload-processing"
Oct 30 17:11:02.199043 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.198909    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" containerName="payload-processing"
Oct 30 17:11:02.199160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.199138    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" containerName="payload-processing"
Oct 30 17:11:02.204257 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.204237    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.211078 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.211054    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"]
Oct 30 17:11:02.221632 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.221597    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-abda73-openshift-default-dockercfg-wwgz9\""
Oct 30 17:11:02.306874 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.306833    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8a3e32-72bb-463b-8cc6-93999ce0d2a4" path="/var/lib/kubelet/pods/4d8a3e32-72bb-463b-8cc6-93999ce0d2a4/volumes"
Oct 30 17:11:02.307398 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.307381    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece549bc-d032-487d-8c0b-466e49eae5fb" path="/var/lib/kubelet/pods/ece549bc-d032-487d-8c0b-466e49eae5fb/volumes"
Oct 30 17:11:02.315798 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.315768    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c597a45-d66f-4231-8243-4867e0850f31-istio-podinfo\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.315931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.315817    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c597a45-d66f-4231-8243-4867e0850f31-istiod-ca-cert\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.315970 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.315930    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-envoy\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.316003 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.315975    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.316047 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.316013    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-credential-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.316128 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.316102    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-istio-token\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.316251 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.316234    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t85tn\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-kube-api-access-t85tn\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.316311 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.316273    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-certs\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.316399 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.316380    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-data\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.347511 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.347481    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:11:02.408317 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.408285    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:11:02.408829 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.408340    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:11:02.417893 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.417861    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-data\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418067 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.417945    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c597a45-d66f-4231-8243-4867e0850f31-istio-podinfo\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418067 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.417998    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c597a45-d66f-4231-8243-4867e0850f31-istiod-ca-cert\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418251 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.418080    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-envoy\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418251 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.418165    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418251 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.418238    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-credential-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418433 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.418277    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-istio-token\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418433 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.418338    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t85tn\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-kube-api-access-t85tn\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418433 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.418387    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-certs\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418611 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.418476    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-data\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.418765 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.418739    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-credential-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.419450 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.419421    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-certs\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.419579 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.419563    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.419652 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.419570    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c597a45-d66f-4231-8243-4867e0850f31-istiod-ca-cert\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.422001 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.421968    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c597a45-d66f-4231-8243-4867e0850f31-istio-podinfo\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.429664 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.429629    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-envoy\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.429906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.429880    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"]
Oct 30 17:11:02.442650 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.442616    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-istio-token\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.444042 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.443778    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t85tn\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-kube-api-access-t85tn\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.522472 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.522439    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:02.904993 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.904965    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"]
Oct 30 17:11:02.906195 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:11:02.906155    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c597a45_d66f_4231_8243_4867e0850f31.slice/crio-5cd4e3462e11aa04ce6c5e94524faaf36ba7dcc023aa8acc479e7fc06e026b75 WatchSource:0}: Error finding container 5cd4e3462e11aa04ce6c5e94524faaf36ba7dcc023aa8acc479e7fc06e026b75: Status 404 returned error can't find the container with id 5cd4e3462e11aa04ce6c5e94524faaf36ba7dcc023aa8acc479e7fc06e026b75
Oct 30 17:11:02.909124 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.909090    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:02.909240 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.909154    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:02.909240 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:02.909202    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:03.404966 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.404924    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:11:03.405221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.404996    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:11:03.412804 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.412674    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"]
Oct 30 17:11:03.419304 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.419276    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"]
Oct 30 17:11:03.419492 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.419390    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.432054 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.432015    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-ckw59\""
Oct 30 17:11:03.433377 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.433354    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.446362 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.446332    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"]
Oct 30 17:11:03.449702 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.449630    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"]
Oct 30 17:11:03.508626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.508587    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.508626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.508627    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.508821 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.508698    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.509178 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.509148    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.509296 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.509277    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.509378 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.509362    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.509418 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.509399    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltsqd\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-kube-api-access-ltsqd\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.509496 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.509481    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.509533 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.509508    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.515775 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.515735    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" event={"ID":"3c597a45-d66f-4231-8243-4867e0850f31","Type":"ContainerStarted","Data":"07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23"}
Oct 30 17:11:03.515908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.515805    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" event={"ID":"3c597a45-d66f-4231-8243-4867e0850f31","Type":"ContainerStarted","Data":"5cd4e3462e11aa04ce6c5e94524faaf36ba7dcc023aa8acc479e7fc06e026b75"}
Oct 30 17:11:03.522882 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.522854    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:03.587213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.574522    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" podStartSLOduration=1.574501353 podStartE2EDuration="1.574501353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:03.569999981 +0000 UTC m=+2893.423878259" watchObservedRunningTime="2026-10-30 17:11:03.574501353 +0000 UTC m=+2893.428379619"
Oct 30 17:11:03.611092 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.611045    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.611805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.611775    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltsqd\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-kube-api-access-ltsqd\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.611933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.611822    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.611933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.611852    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.611933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.611906    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.612162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.611934    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.612162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.611957    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.612162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.611986    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.612162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612052    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.612162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612086    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06ac1a-0119-4dde-a02a-8ab491565a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.612162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612134    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.612671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612163    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.612671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612233    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.612671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612269    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.612671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612382    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.612671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612439    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.612671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612495    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxcx\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-kube-api-access-zrxcx\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.612671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.612584    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.613830 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.613653    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.614095 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.614074    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.614989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.614938    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.614989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.614950    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.616396 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.616339    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.616869 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.616810    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.619237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.619180    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.632816 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.632769    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.635134 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.634856    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltsqd\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-kube-api-access-ltsqd\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.713816 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.713703    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxcx\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-kube-api-access-zrxcx\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.714064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.713867    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.714064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.713899    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.714064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.713951    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.714064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.713983    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.714064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.714045    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.714370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.714082    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06ac1a-0119-4dde-a02a-8ab491565a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.714370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.714152    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.714370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.714231    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.716373 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.715442    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.716373 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.715772    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06ac1a-0119-4dde-a02a-8ab491565a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.716373 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.716046    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.716373 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.716064    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.716638 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.716458    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.717687 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.717662    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.736314 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.735595    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.739118 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.739091    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.739438 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.739409    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxcx\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-kube-api-access-zrxcx\") pod \"maas-default-gateway-openshift-default-7854949c9-x8n9n\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:03.764833 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.764806    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:03.819852 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:03.819671    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:04.123621 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.122420    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"]
Oct 30 17:11:04.138281 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.137737    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"]
Oct 30 17:11:04.138281 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.137846    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.154356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154115    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.154356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154179    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.154356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154324    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd4t6\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-kube-api-access-xd4t6\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.155376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154706    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.155376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154761    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.155376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154820    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.155376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154851    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.155376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154878    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.155376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.154904    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.156840 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.156815    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2d365c-openshift-default-dockercfg-gctmp\""
Oct 30 17:11:04.242738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.242702    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"]
Oct 30 17:11:04.256566 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.256530    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd4t6\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-kube-api-access-xd4t6\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.256728 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.256638    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.257098 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.257074    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.257558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.257165    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.257662 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.257530    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.257801 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.257783    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.257872 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.257846    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.257926 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.257888    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.257992 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.257931    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.258056 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.258022    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.258111 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.258078    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.258174 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.258152    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.258759 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.258731    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.259285 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.259241    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.261241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.261216    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.262363 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.262341    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.337875 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.337784    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd4t6\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-kube-api-access-xd4t6\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.338890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.338846    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.344535 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.344093    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"]
Oct 30 17:11:04.346060 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:11:04.346034    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c06ac1a_0119_4dde_a02a_8ab491565a05.slice/crio-491ff791c6b2aa04584bd79c4426712df219ff4e14ebbc262b5eeab908a389d8 WatchSource:0}: Error finding container 491ff791c6b2aa04584bd79c4426712df219ff4e14ebbc262b5eeab908a389d8: Status 404 returned error can't find the container with id 491ff791c6b2aa04584bd79c4426712df219ff4e14ebbc262b5eeab908a389d8
Oct 30 17:11:04.348277 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:11:04.348256    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b6e4b9_63df_4089_86b2_7fdb181c1e78.slice/crio-3b2e0b2e064da27bca7a7ad4845f7d5bed05323b66abf6d79b369313c741c22d WatchSource:0}: Error finding container 3b2e0b2e064da27bca7a7ad4845f7d5bed05323b66abf6d79b369313c741c22d: Status 404 returned error can't find the container with id 3b2e0b2e064da27bca7a7ad4845f7d5bed05323b66abf6d79b369313c741c22d
Oct 30 17:11:04.350220 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.350141    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.350292 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.350221    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.350292 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.350252    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.351446 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.351409    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.351541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.351485    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.351541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.351528    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.355694 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.353108    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"]
Oct 30 17:11:04.428421 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.428386    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:11:04.428973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.428943    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:11:04.460251 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.460215    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:04.533090 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.532333    4497 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 17:11:04.533090 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.532380    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 17:11:04.539018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.538978    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd" event={"ID":"48b6e4b9-63df-4089-86b2-7fdb181c1e78","Type":"ContainerStarted","Data":"3b2e0b2e064da27bca7a7ad4845f7d5bed05323b66abf6d79b369313c741c22d"}
Oct 30 17:11:04.541318 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.541286    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" event={"ID":"8c06ac1a-0119-4dde-a02a-8ab491565a05","Type":"ContainerStarted","Data":"491ff791c6b2aa04584bd79c4426712df219ff4e14ebbc262b5eeab908a389d8"}
Oct 30 17:11:04.953460 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.953286    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.953460 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.953382    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.953460 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.953446    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:04.959519 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:04.959462    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"]
Oct 30 17:11:05.407215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.406739    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 17:11:05.407215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.406797    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 17:11:05.537214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.536232    4497 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 17:11:05.537214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.536287    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 17:11:05.565219 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.565163    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd" event={"ID":"48b6e4b9-63df-4089-86b2-7fdb181c1e78","Type":"ContainerStarted","Data":"c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7"}
Oct 30 17:11:05.647469 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.645240    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" event={"ID":"8c06ac1a-0119-4dde-a02a-8ab491565a05","Type":"ContainerStarted","Data":"9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400"}
Oct 30 17:11:05.650012 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.648274    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd" podStartSLOduration=2.6482563519999998 podStartE2EDuration="2.648256352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:05.642945338 +0000 UTC m=+2895.496823603" watchObservedRunningTime="2026-10-30 17:11:05.648256352 +0000 UTC m=+2895.502134621"
Oct 30 17:11:05.650012 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.649887    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" event={"ID":"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9","Type":"ContainerStarted","Data":"01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810"}
Oct 30 17:11:05.650012 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.649945    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" event={"ID":"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9","Type":"ContainerStarted","Data":"0932fe52e86df4fa36691716620ada6470c9366326a65e08dfb246429f61eb1e"}
Oct 30 17:11:05.761081 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.761015    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podStartSLOduration=2.760993299 podStartE2EDuration="2.760993299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:05.706674998 +0000 UTC m=+2895.560553263" watchObservedRunningTime="2026-10-30 17:11:05.760993299 +0000 UTC m=+2895.614871565"
Oct 30 17:11:05.770569 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.769445    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:05.790354 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.771641    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podStartSLOduration=1.771624162 podStartE2EDuration="1.771624162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:05.769627066 +0000 UTC m=+2895.623505344" watchObservedRunningTime="2026-10-30 17:11:05.771624162 +0000 UTC m=+2895.625502427"
Oct 30 17:11:05.822218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.817468    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:05.850172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.835216    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:05.850172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.846407    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x8n9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 17:11:05.850172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:05.846463    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 17:11:06.445269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.434555    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:11:06.445269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.436359    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:11:06.449513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.449426    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:11:06.461485 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.461456    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:06.463400 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.463376    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 17:11:06.463502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.463429    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 17:11:06.532236 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.524275    4497 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 17:11:06.532236 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.524335    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 17:11:06.727271 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.707798    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:06.750214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.742033    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:11:06.837210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.821638    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x8n9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 17:11:06.837210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:06.821700    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 17:11:07.431536 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.431492    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f" podUID="5879e29e-6852-4eee-a56f-6a662d5f32ee" containerName="payload-processing" containerID="cri-o://a18fedbfe45a2632713ad03498bc6a002f90000dbd33d496684d20acff8502ef" gracePeriod=25
Oct 30 17:11:07.461155 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.461116    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 17:11:07.461371 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.461181    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 17:11:07.546214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.542306    4497 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 17:11:07.546214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.542368    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 17:11:07.750289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.747475    4497 generic.go:356] "Generic (PLEG): container finished" podID="5879e29e-6852-4eee-a56f-6a662d5f32ee" containerID="a18fedbfe45a2632713ad03498bc6a002f90000dbd33d496684d20acff8502ef" exitCode=0
Oct 30 17:11:07.750289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.748542    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f" event={"ID":"5879e29e-6852-4eee-a56f-6a662d5f32ee","Type":"ContainerDied","Data":"a18fedbfe45a2632713ad03498bc6a002f90000dbd33d496684d20acff8502ef"}
Oct 30 17:11:07.869513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.868520    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x8n9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 17:11:07.869513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:07.868579    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 17:11:08.165138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.164987    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:11:08.169883 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.169855    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-kube-api-access-vgmvs\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-kube-api-access-vgmvs\") pod \"5879e29e-6852-4eee-a56f-6a662d5f32ee\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") "
Oct 30 17:11:08.170022 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.169956    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-routing-overlay-volume\") pod \"5879e29e-6852-4eee-a56f-6a662d5f32ee\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") "
Oct 30 17:11:08.170315 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.170294    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-routing-overlay-volume" pod "5879e29e-6852-4eee-a56f-6a662d5f32ee" (UID: "5879e29e-6852-4eee-a56f-6a662d5f32ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:11:08.170357 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.170331    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-plugins-config-volume\") pod \"5879e29e-6852-4eee-a56f-6a662d5f32ee\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") "
Oct 30 17:11:08.170536 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.170517    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-provider-credentials\") pod \"5879e29e-6852-4eee-a56f-6a662d5f32ee\" (UID: \"5879e29e-6852-4eee-a56f-6a662d5f32ee\") "
Oct 30 17:11:08.170602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.170580    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-plugins-config-volume" pod "5879e29e-6852-4eee-a56f-6a662d5f32ee" (UID: "5879e29e-6852-4eee-a56f-6a662d5f32ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:11:08.171126 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.171107    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:08.171261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.171130    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5879e29e-6852-4eee-a56f-6a662d5f32ee-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:08.258519 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.258480    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-provider-credentials" pod "5879e29e-6852-4eee-a56f-6a662d5f32ee" (UID: "5879e29e-6852-4eee-a56f-6a662d5f32ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:11:08.258785 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.258757    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-kube-api-access-vgmvs" pod "5879e29e-6852-4eee-a56f-6a662d5f32ee" (UID: "5879e29e-6852-4eee-a56f-6a662d5f32ee"). InnerVolumeSpecName "kube-api-access-vgmvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:11:08.361039 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.360382    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgmvs\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-kube-api-access-vgmvs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:08.361039 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.360420    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5879e29e-6852-4eee-a56f-6a662d5f32ee-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:08.463214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.462416    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 17:11:08.463214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.462476    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 17:11:08.562261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.561714    4497 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 17:11:08.562261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.561775    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 17:11:08.765454 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.765411    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"]
Oct 30 17:11:08.767274 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.767223    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5879e29e-6852-4eee-a56f-6a662d5f32ee" containerName="payload-processing"
Oct 30 17:11:08.767470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.767454    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5879e29e-6852-4eee-a56f-6a662d5f32ee" containerName="payload-processing"
Oct 30 17:11:08.767776 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.767758    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5879e29e-6852-4eee-a56f-6a662d5f32ee" containerName="payload-processing"
Oct 30 17:11:08.923265 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.829020    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x8n9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 17:11:08.923265 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.829085    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 17:11:08.923800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.923760    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"]
Oct 30 17:11:08.949317 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.949282    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"]
Oct 30 17:11:08.988444 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:08.980756    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:09.002155 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.001115    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6acc78\""
Oct 30 17:11:09.004726 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.004703    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:09.014960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.014933    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6acc78-dockercfg-22t62\""
Oct 30 17:11:09.104224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.096271    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fx6d\" (UniqueName: \"kubernetes.io/projected/be939ad3-346e-4401-a5d8-65d4f94d46aa-kube-api-access-5fx6d\") pod \"payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4\" (UID: \"be939ad3-346e-4401-a5d8-65d4f94d46aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:09.104224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.096373    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be939ad3-346e-4401-a5d8-65d4f94d46aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4\" (UID: \"be939ad3-346e-4401-a5d8-65d4f94d46aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:09.104224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.097410    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"]
Oct 30 17:11:09.154208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.148433    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f" event={"ID":"5879e29e-6852-4eee-a56f-6a662d5f32ee","Type":"ContainerDied","Data":"0bbf903e014948ebaf7263c4875e159d1efd3ea6fae1cf9a45a4a8b85a8c9b41"}
Oct 30 17:11:09.154208 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.148525    4497 scope.go:122] "RemoveContainer" containerID="a18fedbfe45a2632713ad03498bc6a002f90000dbd33d496684d20acff8502ef"
Oct 30 17:11:09.168674 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.168640    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"
Oct 30 17:11:09.242367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.202316    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fx6d\" (UniqueName: \"kubernetes.io/projected/be939ad3-346e-4401-a5d8-65d4f94d46aa-kube-api-access-5fx6d\") pod \"payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4\" (UID: \"be939ad3-346e-4401-a5d8-65d4f94d46aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:09.242367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.202422    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be939ad3-346e-4401-a5d8-65d4f94d46aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4\" (UID: \"be939ad3-346e-4401-a5d8-65d4f94d46aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:09.242367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.202502    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw\" (UID: \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:09.242367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.202639    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfldz\" (UniqueName: \"kubernetes.io/projected/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-kube-api-access-pfldz\") pod \"payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw\" (UID: \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:09.242367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.205864    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be939ad3-346e-4401-a5d8-65d4f94d46aa-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4\" (UID: \"be939ad3-346e-4401-a5d8-65d4f94d46aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:09.245303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.245259    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl" podUID="2f1bed12-0da3-45cd-a2d6-01b81d3c480f" containerName="payload-processing" containerID="cri-o://0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389" gracePeriod=25
Oct 30 17:11:09.270568 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.270504    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fx6d\" (UniqueName: \"kubernetes.io/projected/be939ad3-346e-4401-a5d8-65d4f94d46aa-kube-api-access-5fx6d\") pod \"payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4\" (UID: \"be939ad3-346e-4401-a5d8-65d4f94d46aa\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:09.320466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.315286    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw\" (UID: \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:09.320466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.315437    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfldz\" (UniqueName: \"kubernetes.io/projected/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-kube-api-access-pfldz\") pod \"payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw\" (UID: \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:09.320466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.317092    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw\" (UID: \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:09.351667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.325693    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:09.351667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.340468    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"]
Oct 30 17:11:09.351667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.347445    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.396589 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.386317    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-n6ck9\""
Oct 30 17:11:09.425900 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.425864    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfldz\" (UniqueName: \"kubernetes.io/projected/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-kube-api-access-pfldz\") pod \"payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw\" (UID: \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:09.435218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.428935    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"]
Oct 30 17:11:09.437274 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.437234    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"]
Oct 30 17:11:09.500260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.474250    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-6c995dl9f"]
Oct 30 17:11:09.500260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.474678    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 17:11:09.500260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.474741    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 17:11:09.500260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.492368    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.527496    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75j6d\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-kube-api-access-75j6d\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.527557    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.527608    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.527642    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.527676    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.527702    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.527768    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.527820    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.528443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.528156    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.540752 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.540311    4497 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 17:11:09.540752 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.540370    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.632452    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75j6d\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-kube-api-access-75j6d\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.632534    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.632586    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.632623    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.632660    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.632689    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.632758    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.632816    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.633153    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.633737    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.634367    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.634886    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.635148    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.643225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.639460    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.650949 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.650921    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.659707 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.658725    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.665243 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.661327    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.665243 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.661935    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75j6d\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-kube-api-access-75j6d\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789prr7t\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.754957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.754490    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:09.835897 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.835416    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x8n9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 17:11:09.835897 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:09.835478    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 17:11:10.070992 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.070963    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:11:10.138532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.138499    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"]
Oct 30 17:11:10.138783 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.138749    4497 generic.go:356] "Generic (PLEG): container finished" podID="2f1bed12-0da3-45cd-a2d6-01b81d3c480f" containerID="0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389" exitCode=0
Oct 30 17:11:10.138873 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.138850    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"
Oct 30 17:11:10.138873 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.138855    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl" event={"ID":"2f1bed12-0da3-45cd-a2d6-01b81d3c480f","Type":"ContainerDied","Data":"0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389"}
Oct 30 17:11:10.138998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.138900    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl" event={"ID":"2f1bed12-0da3-45cd-a2d6-01b81d3c480f","Type":"ContainerDied","Data":"79746f3af8129778ab7bc4fe3fd95c19a8ed61307d5a2c61360c9788b185cf19"}
Oct 30 17:11:10.138998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.138931    4497 scope.go:122] "RemoveContainer" containerID="0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389"
Oct 30 17:11:10.145558 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:11:10.145527    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe939ad3_346e_4401_a5d8_65d4f94d46aa.slice/crio-c51f87a153588a155cdea2e982c5a369b6756466a3b09195383cd54e21d19c72 WatchSource:0}: Error finding container c51f87a153588a155cdea2e982c5a369b6756466a3b09195383cd54e21d19c72: Status 404 returned error can't find the container with id c51f87a153588a155cdea2e982c5a369b6756466a3b09195383cd54e21d19c72
Oct 30 17:11:10.161453 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.161424    4497 scope.go:122] "RemoveContainer" containerID="0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389"
Oct 30 17:11:10.161957 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:11:10.161922    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389\": container with ID starting with 0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389 not found: ID does not exist" containerID="0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389"
Oct 30 17:11:10.162074 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.161953    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389"} err="failed to get container status \"0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389\": rpc error: code = NotFound desc = could not find container \"0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389\": container with ID starting with 0ca29856e4dd1c3d808b755292c9e7f9367509766dee0612be796ba084304389 not found: ID does not exist"
Oct 30 17:11:10.269278 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.269247    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-kube-api-access-fng45\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-kube-api-access-fng45\") pod \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") "
Oct 30 17:11:10.269454 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.269369    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-provider-credentials\") pod \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") "
Oct 30 17:11:10.269454 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.269405    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-routing-overlay-volume\") pod \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") "
Oct 30 17:11:10.269569 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.269460    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-plugins-config-volume\") pod \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\" (UID: \"2f1bed12-0da3-45cd-a2d6-01b81d3c480f\") "
Oct 30 17:11:10.270257 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.270222    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-routing-overlay-volume" pod "2f1bed12-0da3-45cd-a2d6-01b81d3c480f" (UID: "2f1bed12-0da3-45cd-a2d6-01b81d3c480f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:11:10.270257 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.270244    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-plugins-config-volume" pod "2f1bed12-0da3-45cd-a2d6-01b81d3c480f" (UID: "2f1bed12-0da3-45cd-a2d6-01b81d3c480f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:11:10.328401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.328303    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-kube-api-access-fng45" pod "2f1bed12-0da3-45cd-a2d6-01b81d3c480f" (UID: "2f1bed12-0da3-45cd-a2d6-01b81d3c480f"). InnerVolumeSpecName "kube-api-access-fng45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:11:10.332584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.332554    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-provider-credentials" pod "2f1bed12-0da3-45cd-a2d6-01b81d3c480f" (UID: "2f1bed12-0da3-45cd-a2d6-01b81d3c480f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:11:10.339599 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.339567    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5879e29e-6852-4eee-a56f-6a662d5f32ee" path="/var/lib/kubelet/pods/5879e29e-6852-4eee-a56f-6a662d5f32ee/volumes"
Oct 30 17:11:10.411211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.403748    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:10.411211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.403790    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:10.411211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.403806    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:10.411211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.403821    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fng45\" (UniqueName: \"kubernetes.io/projected/2f1bed12-0da3-45cd-a2d6-01b81d3c480f-kube-api-access-fng45\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:11:10.435724 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.435692    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"]
Oct 30 17:11:10.462002 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.461968    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 17:11:10.462146 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.462027    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 17:11:10.535575 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.534024    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:10.539245 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.536613    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:10.542559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.542444    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:11:10.542559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.542533    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"]
Oct 30 17:11:10.544279 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.544244    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:10.544395 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.544318    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:10.544395 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.544349    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:11:10.547703 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.547683    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-57496rnwl"]
Oct 30 17:11:10.553410 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.553386    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"]
Oct 30 17:11:10.837214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.837094    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x8n9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 17:11:10.837214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:10.837141    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 17:11:11.149728 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.149694    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw" event={"ID":"5a95d1b9-44a3-4292-a1bf-b2e974fffe81","Type":"ContainerStarted","Data":"8ce1988e9d750fd13db3eee12e0bf854ebb11b808baa2b36101254a58632f97d"}
Oct 30 17:11:11.149967 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.149954    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:11.150095 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.150067    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw" event={"ID":"5a95d1b9-44a3-4292-a1bf-b2e974fffe81","Type":"ContainerStarted","Data":"79301d8792fe370e1af5f2fa8d986baffde3d0fd85ef64e4984ee5d1d6a76002"}
Oct 30 17:11:11.152234 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.152213    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4" event={"ID":"be939ad3-346e-4401-a5d8-65d4f94d46aa","Type":"ContainerStarted","Data":"60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e"}
Oct 30 17:11:11.152412 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.152396    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4" event={"ID":"be939ad3-346e-4401-a5d8-65d4f94d46aa","Type":"ContainerStarted","Data":"c51f87a153588a155cdea2e982c5a369b6756466a3b09195383cd54e21d19c72"}
Oct 30 17:11:11.152528 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.152515    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:11.154406 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.154380    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t" event={"ID":"3df30ece-9a53-4a35-be67-d8e6675d6ba8","Type":"ContainerStarted","Data":"bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492"}
Oct 30 17:11:11.154484 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.154430    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t" event={"ID":"3df30ece-9a53-4a35-be67-d8e6675d6ba8","Type":"ContainerStarted","Data":"7c6bf97958c41447f107cbe9266502aedc130778688557070d42a38106dd8edc"}
Oct 30 17:11:11.194222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.193941    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw" podStartSLOduration=3.193918566 podStartE2EDuration="3.193918566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:11.191607158 +0000 UTC m=+2901.045485422" watchObservedRunningTime="2026-10-30 17:11:11.193918566 +0000 UTC m=+2901.047796832"
Oct 30 17:11:11.350923 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.350362    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t" podStartSLOduration=2.350339469 podStartE2EDuration="2.350339469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:11.310168444 +0000 UTC m=+2901.164046710" watchObservedRunningTime="2026-10-30 17:11:11.350339469 +0000 UTC m=+2901.204217735"
Oct 30 17:11:11.423269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.408432    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4" podStartSLOduration=3.408409322 podStartE2EDuration="3.408409322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:11.406829894 +0000 UTC m=+2901.260708163" watchObservedRunningTime="2026-10-30 17:11:11.408409322 +0000 UTC m=+2901.262287588"
Oct 30 17:11:11.461367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.461328    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 17:11:11.461627 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.461596    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 17:11:11.756425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.756387    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:11.762682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.762652    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:11.820897 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.820859    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x8n9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 17:11:11.821088 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:11.820922    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 17:11:12.163449 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:12.163325    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:12.164310 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:12.164282    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:11:12.303973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:12.303938    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1bed12-0da3-45cd-a2d6-01b81d3c480f" path="/var/lib/kubelet/pods/2f1bed12-0da3-45cd-a2d6-01b81d3c480f/volumes"
Oct 30 17:11:12.460812 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:12.460721    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:11:12.460812 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:12.460739    4497 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 17:11:12.460812 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:12.460794    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 17:11:12.821086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:12.821057    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x8n9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 17:11:12.821327 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:12.821114    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 17:11:13.466308 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:13.466273    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:13.466807 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:13.466777    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:13.467647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:13.467627    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:11:14.203172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:14.203140    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:14.203589 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:14.203565    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:14.204467 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:14.204441    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:11:21.918835 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:11:21.918786    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:11:22.166568 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:22.166537    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:11:22.166772 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:22.166755    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:11:32.166334 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:32.166294    4497 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6q6mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 17:11:32.166875 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:32.166360    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 17:11:51.919337 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:11:51.919253    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:11:53.442389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.442360    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"]
Oct 30 17:11:53.443132 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.443117    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2f1bed12-0da3-45cd-a2d6-01b81d3c480f" containerName="payload-processing"
Oct 30 17:11:53.443132 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.443132    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1bed12-0da3-45cd-a2d6-01b81d3c480f" containerName="payload-processing"
Oct 30 17:11:53.443272 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.443257    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1bed12-0da3-45cd-a2d6-01b81d3c480f" containerName="payload-processing"
Oct 30 17:11:53.448165 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.448149    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.465291 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.465258    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6acc78-serving-cert\""
Oct 30 17:11:53.481312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.481222    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"]
Oct 30 17:11:53.554088 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.544242    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153210bf-399a-41bd-a5e3-de2a38b7d549-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.554088 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.544415    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfrj\" (UniqueName: \"kubernetes.io/projected/153210bf-399a-41bd-a5e3-de2a38b7d549-kube-api-access-5vfrj\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.554088 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.544553    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-metrics-certs\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.554088 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.544643    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-maas-api-tls\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.654608 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.654554    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfrj\" (UniqueName: \"kubernetes.io/projected/153210bf-399a-41bd-a5e3-de2a38b7d549-kube-api-access-5vfrj\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.655032 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.654997    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-metrics-certs\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.655169 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.655104    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-maas-api-tls\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.668389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.655300    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153210bf-399a-41bd-a5e3-de2a38b7d549-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.696658 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.674284    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153210bf-399a-41bd-a5e3-de2a38b7d549-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.696658 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.690562    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-maas-api-tls\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.696658 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.692362    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfrj\" (UniqueName: \"kubernetes.io/projected/153210bf-399a-41bd-a5e3-de2a38b7d549-kube-api-access-5vfrj\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.698016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.697986    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-metrics-certs\") pod \"maas-api-e2e-inf-a-6acc78-76865d574-njzw6\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:53.762156 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:53.762119    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:54.071253 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:54.071225    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"]
Oct 30 17:11:54.073958 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:11:54.073923    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153210bf_399a_41bd_a5e3_de2a38b7d549.slice/crio-ccc8ee83cf6a1f4f009c43594fe9f67e612a73586530afdeb7219b5a0829565c WatchSource:0}: Error finding container ccc8ee83cf6a1f4f009c43594fe9f67e612a73586530afdeb7219b5a0829565c: Status 404 returned error can't find the container with id ccc8ee83cf6a1f4f009c43594fe9f67e612a73586530afdeb7219b5a0829565c
Oct 30 17:11:54.075688 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:54.075668    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:11:54.544355 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:54.544322    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6" event={"ID":"153210bf-399a-41bd-a5e3-de2a38b7d549","Type":"ContainerStarted","Data":"ccc8ee83cf6a1f4f009c43594fe9f67e612a73586530afdeb7219b5a0829565c"}
Oct 30 17:11:55.554672 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:55.554636    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6" event={"ID":"153210bf-399a-41bd-a5e3-de2a38b7d549","Type":"ContainerStarted","Data":"dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241"}
Oct 30 17:11:55.555240 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:55.554711    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:11:55.596994 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:11:55.596926    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6" podStartSLOduration=2.157159641 podStartE2EDuration="2.59690922s" totalImagesPullingTime="439.749579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:55.593431888 +0000 UTC m=+2945.447310154" watchObservedRunningTime="2026-10-30 17:11:55.59690922 +0000 UTC m=+2945.450787487"
Oct 30 17:12:01.586592 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:01.586554    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:12:21.918702 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:12:21.918664    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:12:51.107732 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.107700    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"]
Oct 30 17:12:51.112632 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.112607    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:51.139718 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.139683    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-24acb2-dockercfg-t2lpr\""
Oct 30 17:12:51.139853 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.139770    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-24acb2\""
Oct 30 17:12:51.144213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.144172    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"]
Oct 30 17:12:51.222134 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.222096    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5ss\" (UniqueName: \"kubernetes.io/projected/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-kube-api-access-xm5ss\") pod \"payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d\" (UID: \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:51.222377 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.222332    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d\" (UID: \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:51.324480 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.324427    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5ss\" (UniqueName: \"kubernetes.io/projected/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-kube-api-access-xm5ss\") pod \"payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d\" (UID: \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:51.324811 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.324686    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d\" (UID: \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:51.327101 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.326148    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d\" (UID: \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:51.392303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.386489    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5ss\" (UniqueName: \"kubernetes.io/projected/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-kube-api-access-xm5ss\") pod \"payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d\" (UID: \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:51.439363 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.439266    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:51.749261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:51.749238    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"]
Oct 30 17:12:51.751865 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:12:51.751837    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1151b05d_9e88_46f5_9e73_2c7b7b8e8829.slice/crio-ea84552cca805f3f2ae10a6f4d578e1c073d733e2ee1458f271236dbdab9d65e WatchSource:0}: Error finding container ea84552cca805f3f2ae10a6f4d578e1c073d733e2ee1458f271236dbdab9d65e: Status 404 returned error can't find the container with id ea84552cca805f3f2ae10a6f4d578e1c073d733e2ee1458f271236dbdab9d65e
Oct 30 17:12:51.917228 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:12:51.917181    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:12:52.063410 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:52.063370    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d" event={"ID":"1151b05d-9e88-46f5-9e73-2c7b7b8e8829","Type":"ContainerStarted","Data":"f3a4ab434bf065b760588c12bc963f8461f256205115f95b15fc1e3c090fb9b2"}
Oct 30 17:12:52.063620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:52.063437    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:12:52.063620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:52.063465    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d" event={"ID":"1151b05d-9e88-46f5-9e73-2c7b7b8e8829","Type":"ContainerStarted","Data":"ea84552cca805f3f2ae10a6f4d578e1c073d733e2ee1458f271236dbdab9d65e"}
Oct 30 17:12:52.101174 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:12:52.101119    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d" podStartSLOduration=1.10110487 podStartE2EDuration="1.10110487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:52.099794479 +0000 UTC m=+3001.953672745" watchObservedRunningTime="2026-10-30 17:12:52.10110487 +0000 UTC m=+3001.954983134"
Oct 30 17:13:03.073689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:03.073606    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:13:21.918798 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:13:21.918754    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:13:30.503779 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.503748    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"]
Oct 30 17:13:30.508738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.508716    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.521346 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.521323    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"kube-root-ca.crt\""
Oct 30 17:13:30.521346 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.521335    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"routing-overlay\""
Oct 30 17:13:30.521484 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.521349    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"openshift-service-ca.crt\""
Oct 30 17:13:30.531269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.531242    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"agc-cred-7adf77356401\""
Oct 30 17:13:30.531269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.531256    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"agc-cred-ef2761077b71\""
Oct 30 17:13:30.531431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.531277    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-plugins-e2e-extmod-b-13c495\""
Oct 30 17:13:30.531431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.531290    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-e2e--be0deb1b-dockercfg-k86x2\""
Oct 30 17:13:30.540466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.536289    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"]
Oct 30 17:13:30.587784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.587743    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.587996 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.587947    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgph6\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-kube-api-access-xgph6\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.588071 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.588052    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.588139 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.588114    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.689937 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.689903    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgph6\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-kube-api-access-xgph6\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.690118 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.689985    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.690118 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.690051    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.690655 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.690220    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.690708 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.690665    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.690957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.690936    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.692667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.692647    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.710588 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.710555    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgph6\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-kube-api-access-xgph6\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:30.823620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:30.823525    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:31.134994 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:31.134974    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"]
Oct 30 17:13:31.137665 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:13:31.137633    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ad0006_0371_4e7c_97ad_2f69c095e20a.slice/crio-48935ecf1bce5cbe883434f259da2f2f9365c2f4e7328d0708d9f25eca8d501e WatchSource:0}: Error finding container 48935ecf1bce5cbe883434f259da2f2f9365c2f4e7328d0708d9f25eca8d501e: Status 404 returned error can't find the container with id 48935ecf1bce5cbe883434f259da2f2f9365c2f4e7328d0708d9f25eca8d501e
Oct 30 17:13:31.401592 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:31.401499    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq" event={"ID":"37ad0006-0371-4e7c-97ad-2f69c095e20a","Type":"ContainerStarted","Data":"705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056"}
Oct 30 17:13:31.401592 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:31.401559    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq" event={"ID":"37ad0006-0371-4e7c-97ad-2f69c095e20a","Type":"ContainerStarted","Data":"48935ecf1bce5cbe883434f259da2f2f9365c2f4e7328d0708d9f25eca8d501e"}
Oct 30 17:13:31.401592 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:31.401592    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:31.459063 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:31.459004    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq" podStartSLOduration=1.458988365 podStartE2EDuration="1.458988365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:31.438434595 +0000 UTC m=+3041.292312860" watchObservedRunningTime="2026-10-30 17:13:31.458988365 +0000 UTC m=+3041.312866629"
Oct 30 17:13:42.413276 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:42.413247    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:13:51.373019 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:51.372218    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"]
Oct 30 17:13:51.991208 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:13:51.990854    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:13:52.372282 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:52.372250    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"]
Oct 30 17:13:56.373072 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.373030    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2" podUID="d64df251-52af-4819-ae5b-79db38f8fc0c" containerName="payload-processing" containerID="cri-o://9f198b7d364d77ebbf7dfab9e1ba4b42a556b716d1043246ca82f425358860b7" gracePeriod=25
Oct 30 17:13:56.596612 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.596575    4497 generic.go:356] "Generic (PLEG): container finished" podID="d64df251-52af-4819-ae5b-79db38f8fc0c" containerID="9f198b7d364d77ebbf7dfab9e1ba4b42a556b716d1043246ca82f425358860b7" exitCode=0
Oct 30 17:13:56.596806 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.596649    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2" event={"ID":"d64df251-52af-4819-ae5b-79db38f8fc0c","Type":"ContainerDied","Data":"9f198b7d364d77ebbf7dfab9e1ba4b42a556b716d1043246ca82f425358860b7"}
Oct 30 17:13:56.657300 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.657270    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:13:56.749838 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.749784    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-kube-api-access-w9zn7\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-kube-api-access-w9zn7\") pod \"d64df251-52af-4819-ae5b-79db38f8fc0c\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") "
Oct 30 17:13:56.750034 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.749921    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-plugins-config-volume\") pod \"d64df251-52af-4819-ae5b-79db38f8fc0c\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") "
Oct 30 17:13:56.750034 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.749989    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-provider-credentials\") pod \"d64df251-52af-4819-ae5b-79db38f8fc0c\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") "
Oct 30 17:13:56.750237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.750214    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-routing-overlay-volume\") pod \"d64df251-52af-4819-ae5b-79db38f8fc0c\" (UID: \"d64df251-52af-4819-ae5b-79db38f8fc0c\") "
Oct 30 17:13:56.750712 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.750683    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-plugins-config-volume" pod "d64df251-52af-4819-ae5b-79db38f8fc0c" (UID: "d64df251-52af-4819-ae5b-79db38f8fc0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:56.750872 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.750713    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-routing-overlay-volume" pod "d64df251-52af-4819-ae5b-79db38f8fc0c" (UID: "d64df251-52af-4819-ae5b-79db38f8fc0c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:56.752396 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.752369    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-kube-api-access-w9zn7" pod "d64df251-52af-4819-ae5b-79db38f8fc0c" (UID: "d64df251-52af-4819-ae5b-79db38f8fc0c"). InnerVolumeSpecName "kube-api-access-w9zn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:56.753379 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.753361    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-provider-credentials" pod "d64df251-52af-4819-ae5b-79db38f8fc0c" (UID: "d64df251-52af-4819-ae5b-79db38f8fc0c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:56.852888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.852853    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9zn7\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-kube-api-access-w9zn7\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:13:56.852888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.852885    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:13:56.853087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.852895    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d64df251-52af-4819-ae5b-79db38f8fc0c-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:13:56.853087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:56.852906    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d64df251-52af-4819-ae5b-79db38f8fc0c-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:13:57.372664 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.372617    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4" podUID="08c82c9a-6de8-4ec6-a119-9b1fa332612c" containerName="payload-processing" containerID="cri-o://69a2a11a79f60912368541781df99df4af4999d189d34d12824e156eb61c4399" gracePeriod=25
Oct 30 17:13:57.606305 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.606070    4497 generic.go:356] "Generic (PLEG): container finished" podID="08c82c9a-6de8-4ec6-a119-9b1fa332612c" containerID="69a2a11a79f60912368541781df99df4af4999d189d34d12824e156eb61c4399" exitCode=0
Oct 30 17:13:57.606305 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.606113    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4" event={"ID":"08c82c9a-6de8-4ec6-a119-9b1fa332612c","Type":"ContainerDied","Data":"69a2a11a79f60912368541781df99df4af4999d189d34d12824e156eb61c4399"}
Oct 30 17:13:57.607977 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.607951    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2" event={"ID":"d64df251-52af-4819-ae5b-79db38f8fc0c","Type":"ContainerDied","Data":"e1e6f2deb917e27e713feb57a4b74e87a46ec71b78e6895d0f9faf63473f1d2d"}
Oct 30 17:13:57.608142 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.608009    4497 scope.go:122] "RemoveContainer" containerID="9f198b7d364d77ebbf7dfab9e1ba4b42a556b716d1043246ca82f425358860b7"
Oct 30 17:13:57.608142 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.607959    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"
Oct 30 17:13:57.653757 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.653722    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"]
Oct 30 17:13:57.661266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.661246    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:13:57.662737 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.662716    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9bb5b1/payload-processing-external-model-e2e-extmod-a-9bb5b1-5b748wnr2"]
Oct 30 17:13:57.769553 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.769519    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-kube-api-access-fgtb8\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-kube-api-access-fgtb8\") pod \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") "
Oct 30 17:13:57.769751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.769566    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-plugins-config-volume\") pod \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") "
Oct 30 17:13:57.769751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.769648    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-routing-overlay-volume\") pod \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") "
Oct 30 17:13:57.769751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.769681    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-provider-credentials\") pod \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\" (UID: \"08c82c9a-6de8-4ec6-a119-9b1fa332612c\") "
Oct 30 17:13:57.770111 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.770083    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-routing-overlay-volume" pod "08c82c9a-6de8-4ec6-a119-9b1fa332612c" (UID: "08c82c9a-6de8-4ec6-a119-9b1fa332612c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:57.770259 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.770088    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-plugins-config-volume" pod "08c82c9a-6de8-4ec6-a119-9b1fa332612c" (UID: "08c82c9a-6de8-4ec6-a119-9b1fa332612c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:13:57.770523 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.770506    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:13:57.770569 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.770525    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08c82c9a-6de8-4ec6-a119-9b1fa332612c-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:13:57.771785 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.771757    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-kube-api-access-fgtb8" pod "08c82c9a-6de8-4ec6-a119-9b1fa332612c" (UID: "08c82c9a-6de8-4ec6-a119-9b1fa332612c"). InnerVolumeSpecName "kube-api-access-fgtb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:57.771882 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.771844    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-provider-credentials" pod "08c82c9a-6de8-4ec6-a119-9b1fa332612c" (UID: "08c82c9a-6de8-4ec6-a119-9b1fa332612c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:13:57.872427 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.872392    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtb8\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-kube-api-access-fgtb8\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:13:57.872427 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.872422    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08c82c9a-6de8-4ec6-a119-9b1fa332612c-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:13:57.946962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.946869    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"]
Oct 30 17:13:57.948095 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.948069    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d64df251-52af-4819-ae5b-79db38f8fc0c" containerName="payload-processing"
Oct 30 17:13:57.948095 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.948096    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64df251-52af-4819-ae5b-79db38f8fc0c" containerName="payload-processing"
Oct 30 17:13:57.948344 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.948146    4497 cpu_manager.go:414] "containerMap: removing container" podUID="08c82c9a-6de8-4ec6-a119-9b1fa332612c" containerName="payload-processing"
Oct 30 17:13:57.948344 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.948155    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c82c9a-6de8-4ec6-a119-9b1fa332612c" containerName="payload-processing"
Oct 30 17:13:57.948482 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.948379    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64df251-52af-4819-ae5b-79db38f8fc0c" containerName="payload-processing"
Oct 30 17:13:57.948482 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.948408    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c82c9a-6de8-4ec6-a119-9b1fa332612c" containerName="payload-processing"
Oct 30 17:13:57.952873 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.952853    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:57.968897 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.968872    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6e08bd-openshift-default-dockercfg-cwm9q\""
Oct 30 17:13:57.980607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:57.980581    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"]
Oct 30 17:13:58.075607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.075563    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f459f\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-kube-api-access-f459f\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.075823 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.075724    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-envoy\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.075895 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.075845    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-podinfo\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.075895 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.075876    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-credential-socket\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.076017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.075929    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-token\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.076091 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.076070    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-certs\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.076153 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.076143    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-data\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.076260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.076170    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istiod-ca-cert\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.076322 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.076260    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-socket\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.178564    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-podinfo\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.178621    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-credential-socket\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.178680    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-token\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.178796    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-certs\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.178861    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-data\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.178887    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istiod-ca-cert\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.178945    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-socket\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.179001    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f459f\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-kube-api-access-f459f\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.179129    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-envoy\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.180022    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-certs\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.180682    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-credential-socket\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.181357    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istiod-ca-cert\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.181594    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-data\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.184223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.181842    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-socket\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.190241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.190164    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-envoy\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.195210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.192672    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-podinfo\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.206406 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.206331    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f459f\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-kube-api-access-f459f\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.207732 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.207679    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-token\") pod \"e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") " pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.274229 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.273936    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:13:58.303085 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.303054    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64df251-52af-4819-ae5b-79db38f8fc0c" path="/var/lib/kubelet/pods/d64df251-52af-4819-ae5b-79db38f8fc0c/volumes"
Oct 30 17:13:58.618569 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.618532    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4" event={"ID":"08c82c9a-6de8-4ec6-a119-9b1fa332612c","Type":"ContainerDied","Data":"40889f492adacda7d7193514231d414f1844469b7b869aed9c750bffc572770f"}
Oct 30 17:13:58.618569 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.618562    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"
Oct 30 17:13:58.619157 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.618635    4497 scope.go:122] "RemoveContainer" containerID="69a2a11a79f60912368541781df99df4af4999d189d34d12824e156eb61c4399"
Oct 30 17:13:58.628693 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.628670    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"]
Oct 30 17:13:58.633492 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:13:58.633463    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ee375e_34e5_45d1_b01c_61ce79ef4c33.slice/crio-4bace96c76d3164b5216350892c8ba3d2bd6a52314689469e32c7949d2f18b74 WatchSource:0}: Error finding container 4bace96c76d3164b5216350892c8ba3d2bd6a52314689469e32c7949d2f18b74: Status 404 returned error can't find the container with id 4bace96c76d3164b5216350892c8ba3d2bd6a52314689469e32c7949d2f18b74
Oct 30 17:13:58.636879 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.636847    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:13:58.636988 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.636913    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:13:58.636988 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.636944    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:13:58.671837 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.671811    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"]
Oct 30 17:13:58.677747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:58.677718    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-df67gfln4"]
Oct 30 17:13:59.628149 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:59.628112    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" event={"ID":"f3ee375e-34e5-45d1-b01c-61ce79ef4c33","Type":"ContainerStarted","Data":"b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677"}
Oct 30 17:13:59.628675 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:59.628179    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" event={"ID":"f3ee375e-34e5-45d1-b01c-61ce79ef4c33","Type":"ContainerStarted","Data":"4bace96c76d3164b5216350892c8ba3d2bd6a52314689469e32c7949d2f18b74"}
Oct 30 17:13:59.694113 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:13:59.694052    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" podStartSLOduration=2.6940321369999998 podStartE2EDuration="2.694032137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:59.689747019 +0000 UTC m=+3069.543625285" watchObservedRunningTime="2026-10-30 17:13:59.694032137 +0000 UTC m=+3069.547910403"
Oct 30 17:14:00.274676 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:00.274638    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:14:00.281523 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:00.281497    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:14:00.301696 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:00.301661    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c82c9a-6de8-4ec6-a119-9b1fa332612c" path="/var/lib/kubelet/pods/08c82c9a-6de8-4ec6-a119-9b1fa332612c/volumes"
Oct 30 17:14:00.636094 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:00.636011    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:14:00.636915 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:00.636895    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:14:05.282090 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.282056    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"]
Oct 30 17:14:05.288748 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.288721    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:05.306016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.305985    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6e08bd\""
Oct 30 17:14:05.309258 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.309234    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"]
Oct 30 17:14:05.313721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.313698    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6e08bd-dockercfg-5f7gt\""
Oct 30 17:14:05.440832 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.440798    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnndq\" (UniqueName: \"kubernetes.io/projected/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-kube-api-access-bnndq\") pod \"payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p\" (UID: \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:05.441011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.440897    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p\" (UID: \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:05.563257 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.558709    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnndq\" (UniqueName: \"kubernetes.io/projected/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-kube-api-access-bnndq\") pod \"payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p\" (UID: \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:05.563257 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.558851    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p\" (UID: \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:05.563257 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.559950    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p\" (UID: \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:05.651913 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.651872    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnndq\" (UniqueName: \"kubernetes.io/projected/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-kube-api-access-bnndq\") pod \"payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p\" (UID: \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:05.905058 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:05.904973    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:06.214362 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:06.214286    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"]
Oct 30 17:14:06.218157 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:14:06.218130    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc660447_d7c5_4f66_8fd7_460e4b6f5f8b.slice/crio-04b724f9f0d5fb2bdc1534a6abe1eec5dc2a2d8a304aa4073b0c81974a50aa79 WatchSource:0}: Error finding container 04b724f9f0d5fb2bdc1534a6abe1eec5dc2a2d8a304aa4073b0c81974a50aa79: Status 404 returned error can't find the container with id 04b724f9f0d5fb2bdc1534a6abe1eec5dc2a2d8a304aa4073b0c81974a50aa79
Oct 30 17:14:06.703075 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:06.703039    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p" event={"ID":"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b","Type":"ContainerStarted","Data":"f846818075fee527557a75f4ed189c3769288be7fcc54b29d240b121d173a92a"}
Oct 30 17:14:06.703547 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:06.703102    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p" event={"ID":"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b","Type":"ContainerStarted","Data":"04b724f9f0d5fb2bdc1534a6abe1eec5dc2a2d8a304aa4073b0c81974a50aa79"}
Oct 30 17:14:06.703547 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:06.703215    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:06.747711 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:06.747654    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p" podStartSLOduration=1.747641292 podStartE2EDuration="1.747641292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:06.742513352 +0000 UTC m=+3076.596391619" watchObservedRunningTime="2026-10-30 17:14:06.747641292 +0000 UTC m=+3076.601519556"
Oct 30 17:14:11.071301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.071266    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"]
Oct 30 17:14:11.076742 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.076719    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:11.094804 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.094776    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"]
Oct 30 17:14:11.127220 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.127167    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9afbb2\""
Oct 30 17:14:11.130408 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.130367    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9afbb2-dockercfg-7srnp\""
Oct 30 17:14:11.154514 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.154475    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k274c\" (UniqueName: \"kubernetes.io/projected/b1915fda-de7a-4552-a52d-469a2deb8ac2-kube-api-access-k274c\") pod \"payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz\" (UID: \"b1915fda-de7a-4552-a52d-469a2deb8ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:11.154891 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.154869    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1915fda-de7a-4552-a52d-469a2deb8ac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz\" (UID: \"b1915fda-de7a-4552-a52d-469a2deb8ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:11.297211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.282427    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1915fda-de7a-4552-a52d-469a2deb8ac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz\" (UID: \"b1915fda-de7a-4552-a52d-469a2deb8ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:11.336138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.307899    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k274c\" (UniqueName: \"kubernetes.io/projected/b1915fda-de7a-4552-a52d-469a2deb8ac2-kube-api-access-k274c\") pod \"payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz\" (UID: \"b1915fda-de7a-4552-a52d-469a2deb8ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:11.336138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.299943    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1915fda-de7a-4552-a52d-469a2deb8ac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz\" (UID: \"b1915fda-de7a-4552-a52d-469a2deb8ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:11.347938 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.347908    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k274c\" (UniqueName: \"kubernetes.io/projected/b1915fda-de7a-4552-a52d-469a2deb8ac2-kube-api-access-k274c\") pod \"payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz\" (UID: \"b1915fda-de7a-4552-a52d-469a2deb8ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:11.392025 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.391992    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:11.707154 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.707129    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"]
Oct 30 17:14:11.710064 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:14:11.710035    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1915fda_de7a_4552_a52d_469a2deb8ac2.slice/crio-5e5420ef6972da8addcd639eae6952014ec53274fee12a23064372985df930e5 WatchSource:0}: Error finding container 5e5420ef6972da8addcd639eae6952014ec53274fee12a23064372985df930e5: Status 404 returned error can't find the container with id 5e5420ef6972da8addcd639eae6952014ec53274fee12a23064372985df930e5
Oct 30 17:14:11.743645 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:11.743609    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz" event={"ID":"b1915fda-de7a-4552-a52d-469a2deb8ac2","Type":"ContainerStarted","Data":"5e5420ef6972da8addcd639eae6952014ec53274fee12a23064372985df930e5"}
Oct 30 17:14:12.752544 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:12.752505    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz" event={"ID":"b1915fda-de7a-4552-a52d-469a2deb8ac2","Type":"ContainerStarted","Data":"517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331"}
Oct 30 17:14:12.752968 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:12.752591    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:12.792303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:12.792252    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz" podStartSLOduration=1.792235217 podStartE2EDuration="1.792235217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:12.78829809 +0000 UTC m=+3082.642176355" watchObservedRunningTime="2026-10-30 17:14:12.792235217 +0000 UTC m=+3082.646113517"
Oct 30 17:14:14.313044 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.313014    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"]
Oct 30 17:14:14.317890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.317872    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.330067 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.330047    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 17:14:14.330067 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.330057    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 17:14:14.330235 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.330121    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pmfs7\""
Oct 30 17:14:14.332901 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.332886    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 17:14:14.344255 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.344231    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"]
Oct 30 17:14:14.381038 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.381003    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxgc\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-kube-api-access-ptxgc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.381574 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.381215    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.381574 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.381365    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.381574 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.381559    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/157e4427-efba-47e6-8332-f60180eca846-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.381841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.381674    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.381841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.381705    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.381841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.381778    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/157e4427-efba-47e6-8332-f60180eca846-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.483658 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.483629    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.483846 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.483668    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.483846 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.483734    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/157e4427-efba-47e6-8332-f60180eca846-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.483846 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.483787    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxgc\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-kube-api-access-ptxgc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.484029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.483868    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.484029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.483947    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.484142 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.484092    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/157e4427-efba-47e6-8332-f60180eca846-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.484821 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.484798    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/157e4427-efba-47e6-8332-f60180eca846-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.486177 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.486151    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/157e4427-efba-47e6-8332-f60180eca846-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.486300 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.486255    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.486711 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.486685    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.486796 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.486689    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.504093 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.504060    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.504503 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.504478    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxgc\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-kube-api-access-ptxgc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zbvxm\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.631113 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.631038    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:14.961602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.961570    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"]
Oct 30 17:14:14.962668 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:14:14.962644    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157e4427_efba_47e6_8332_f60180eca846.slice/crio-7922b726efc401b909b4d65bbf465e41f02d06578e0344ab583cd0a9d56ea78f WatchSource:0}: Error finding container 7922b726efc401b909b4d65bbf465e41f02d06578e0344ab583cd0a9d56ea78f: Status 404 returned error can't find the container with id 7922b726efc401b909b4d65bbf465e41f02d06578e0344ab583cd0a9d56ea78f
Oct 30 17:14:14.965484 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.965452    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:14:14.965568 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:14.965515    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:14:15.783413 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:15.783369    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm" event={"ID":"157e4427-efba-47e6-8332-f60180eca846","Type":"ContainerStarted","Data":"d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf"}
Oct 30 17:14:15.783933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:15.783437    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm" event={"ID":"157e4427-efba-47e6-8332-f60180eca846","Type":"ContainerStarted","Data":"7922b726efc401b909b4d65bbf465e41f02d06578e0344ab583cd0a9d56ea78f"}
Oct 30 17:14:15.783933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:15.783679    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:15.847475 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:15.847427    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm" podStartSLOduration=1.847412608 podStartE2EDuration="1.847412608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:15.832663768 +0000 UTC m=+3085.686542034" watchObservedRunningTime="2026-10-30 17:14:15.847412608 +0000 UTC m=+3085.701290874"
Oct 30 17:14:16.792361 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:16.792333    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:14:17.723263 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:17.723225    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:14:21.917881 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:14:21.917844    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:14:23.764789 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:14:23.764757    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:14:51.917635 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:14:51.917601    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:15:00.203039 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.203002    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"]
Oct 30 17:15:00.226179 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.226144    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:00.249762 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.249740    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"]
Oct 30 17:15:00.264148 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.259371    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:00.290747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.290713    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\""
Oct 30 17:15:00.311530 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.311473    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"]
Oct 30 17:15:00.317680 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.317645    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"]
Oct 30 17:15:00.317816 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.317700    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"]
Oct 30 17:15:00.317816 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.317771    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:00.323444 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.323420    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"]
Oct 30 17:15:00.323600 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.323461    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"]
Oct 30 17:15:00.323600 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.323544    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:00.326064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.323792    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"]
Oct 30 17:15:00.337529 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.335495    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"]
Oct 30 17:15:00.337529 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.335580    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:00.340704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.340615    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"]
Oct 30 17:15:00.342565 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.342544    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hp8\" (UniqueName: \"kubernetes.io/projected/a40ddc9f-a668-4958-81a5-cac8f457d495-kube-api-access-48hp8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn\" (UID: \"a40ddc9f-a668-4958-81a5-cac8f457d495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:00.342669 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.342600    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40ddc9f-a668-4958-81a5-cac8f457d495-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn\" (UID: \"a40ddc9f-a668-4958-81a5-cac8f457d495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:00.342874 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.342853    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfh89\" (UniqueName: \"kubernetes.io/projected/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-kube-api-access-kfh89\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm\" (UID: \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:00.342931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.342901    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm\" (UID: \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:00.444308 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.444270    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40ddc9f-a668-4958-81a5-cac8f457d495-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn\" (UID: \"a40ddc9f-a668-4958-81a5-cac8f457d495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:00.444531 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.444335    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59zbb\" (UniqueName: \"kubernetes.io/projected/14ba928c-940f-4a37-af0d-0eef4245519e-kube-api-access-59zbb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7\" (UID: \"14ba928c-940f-4a37-af0d-0eef4245519e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:00.444531 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.444487    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04408a47-037c-41d2-95fa-f97ee17788eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79\" (UID: \"04408a47-037c-41d2-95fa-f97ee17788eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:00.444648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.444588    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfh89\" (UniqueName: \"kubernetes.io/projected/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-kube-api-access-kfh89\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm\" (UID: \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:00.444648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.444618    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm\" (UID: \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:00.444648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.444642    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjbt\" (UniqueName: \"kubernetes.io/projected/ead4d5e1-6dea-496d-bdae-2b24341cdc13-kube-api-access-9hjbt\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct\" (UID: \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:00.444823 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.444753    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead4d5e1-6dea-496d-bdae-2b24341cdc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct\" (UID: \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:00.444891 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.444849    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpdvg\" (UniqueName: \"kubernetes.io/projected/04408a47-037c-41d2-95fa-f97ee17788eb-kube-api-access-mpdvg\") pod \"maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79\" (UID: \"04408a47-037c-41d2-95fa-f97ee17788eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:00.445035 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.445012    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ba928c-940f-4a37-af0d-0eef4245519e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7\" (UID: \"14ba928c-940f-4a37-af0d-0eef4245519e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:00.445090 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.445050    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40ddc9f-a668-4958-81a5-cac8f457d495-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn\" (UID: \"a40ddc9f-a668-4958-81a5-cac8f457d495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:00.445255 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.445235    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hp8\" (UniqueName: \"kubernetes.io/projected/a40ddc9f-a668-4958-81a5-cac8f457d495-kube-api-access-48hp8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn\" (UID: \"a40ddc9f-a668-4958-81a5-cac8f457d495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:00.445351 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.445285    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm\" (UID: \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:00.466972 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.466917    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hp8\" (UniqueName: \"kubernetes.io/projected/a40ddc9f-a668-4958-81a5-cac8f457d495-kube-api-access-48hp8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn\" (UID: \"a40ddc9f-a668-4958-81a5-cac8f457d495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:00.467088 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.467066    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfh89\" (UniqueName: \"kubernetes.io/projected/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-kube-api-access-kfh89\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm\" (UID: \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:00.546684 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.546649    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59zbb\" (UniqueName: \"kubernetes.io/projected/14ba928c-940f-4a37-af0d-0eef4245519e-kube-api-access-59zbb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7\" (UID: \"14ba928c-940f-4a37-af0d-0eef4245519e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:00.546857 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.546736    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04408a47-037c-41d2-95fa-f97ee17788eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79\" (UID: \"04408a47-037c-41d2-95fa-f97ee17788eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:00.546857 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.546778    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjbt\" (UniqueName: \"kubernetes.io/projected/ead4d5e1-6dea-496d-bdae-2b24341cdc13-kube-api-access-9hjbt\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct\" (UID: \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:00.546857 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.546844    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead4d5e1-6dea-496d-bdae-2b24341cdc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct\" (UID: \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:00.546985 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.546867    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpdvg\" (UniqueName: \"kubernetes.io/projected/04408a47-037c-41d2-95fa-f97ee17788eb-kube-api-access-mpdvg\") pod \"maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79\" (UID: \"04408a47-037c-41d2-95fa-f97ee17788eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:00.546985 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.546962    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ba928c-940f-4a37-af0d-0eef4245519e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7\" (UID: \"14ba928c-940f-4a37-af0d-0eef4245519e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:00.547558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.547531    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04408a47-037c-41d2-95fa-f97ee17788eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79\" (UID: \"04408a47-037c-41d2-95fa-f97ee17788eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:00.547674 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.547563    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead4d5e1-6dea-496d-bdae-2b24341cdc13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct\" (UID: \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:00.547674 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.547630    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ba928c-940f-4a37-af0d-0eef4245519e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7\" (UID: \"14ba928c-940f-4a37-af0d-0eef4245519e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:00.567697 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.567671    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpdvg\" (UniqueName: \"kubernetes.io/projected/04408a47-037c-41d2-95fa-f97ee17788eb-kube-api-access-mpdvg\") pod \"maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79\" (UID: \"04408a47-037c-41d2-95fa-f97ee17788eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:00.567806 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.567740    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjbt\" (UniqueName: \"kubernetes.io/projected/ead4d5e1-6dea-496d-bdae-2b24341cdc13-kube-api-access-9hjbt\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct\" (UID: \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:00.567806 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.567751    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59zbb\" (UniqueName: \"kubernetes.io/projected/14ba928c-940f-4a37-af0d-0eef4245519e-kube-api-access-59zbb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7\" (UID: \"14ba928c-940f-4a37-af0d-0eef4245519e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:00.597027 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.597008    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:00.616910 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.616887    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:00.646544 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.646515    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:00.652366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.652203    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:00.666528 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.666362    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:00.923040 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:00.922940    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"]
Oct 30 17:15:00.925350 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:00.925305    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40ddc9f_a668_4958_81a5_cac8f457d495.slice/crio-f8ac7f1913cf214c711bd0b12b963711d346b20077436525dd286bd06bcd489f WatchSource:0}: Error finding container f8ac7f1913cf214c711bd0b12b963711d346b20077436525dd286bd06bcd489f: Status 404 returned error can't find the container with id f8ac7f1913cf214c711bd0b12b963711d346b20077436525dd286bd06bcd489f
Oct 30 17:15:01.048181 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.048153    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"]
Oct 30 17:15:01.050291 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:01.050259    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead4d5e1_6dea_496d_bdae_2b24341cdc13.slice/crio-4536d4acc2e03efb098f3e13c1f66eb2922f6aa46ddb523180a3756a253a1998 WatchSource:0}: Error finding container 4536d4acc2e03efb098f3e13c1f66eb2922f6aa46ddb523180a3756a253a1998: Status 404 returned error can't find the container with id 4536d4acc2e03efb098f3e13c1f66eb2922f6aa46ddb523180a3756a253a1998
Oct 30 17:15:01.087855 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.087813    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"]
Oct 30 17:15:01.089585 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:01.089555    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ba928c_940f_4a37_af0d_0eef4245519e.slice/crio-212ff74aaa0d949899f363cc7f94f235ce05e41d5b923ac7be1404138e90a2f7 WatchSource:0}: Error finding container 212ff74aaa0d949899f363cc7f94f235ce05e41d5b923ac7be1404138e90a2f7: Status 404 returned error can't find the container with id 212ff74aaa0d949899f363cc7f94f235ce05e41d5b923ac7be1404138e90a2f7
Oct 30 17:15:01.092017 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:01.091993    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04408a47_037c_41d2_95fa_f97ee17788eb.slice/crio-7fc02120f7d281d6acbceb4bc04d71e9eeb5d0faf0d9fae18bba4297e69f8273 WatchSource:0}: Error finding container 7fc02120f7d281d6acbceb4bc04d71e9eeb5d0faf0d9fae18bba4297e69f8273: Status 404 returned error can't find the container with id 7fc02120f7d281d6acbceb4bc04d71e9eeb5d0faf0d9fae18bba4297e69f8273
Oct 30 17:15:01.092160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.092141    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"]
Oct 30 17:15:01.159402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.159379    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"]
Oct 30 17:15:01.161377 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:01.161354    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17877f0b_5c58_46f9_9e99_7c4afe23c8e3.slice/crio-591b7f610f46a3cb0ebd72e2b496f9b5d35b134fb3fe28cd088336859bc4d8f7 WatchSource:0}: Error finding container 591b7f610f46a3cb0ebd72e2b496f9b5d35b134fb3fe28cd088336859bc4d8f7: Status 404 returned error can't find the container with id 591b7f610f46a3cb0ebd72e2b496f9b5d35b134fb3fe28cd088336859bc4d8f7
Oct 30 17:15:01.262387 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.262356    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" event={"ID":"17877f0b-5c58-46f9-9e99-7c4afe23c8e3","Type":"ContainerStarted","Data":"591b7f610f46a3cb0ebd72e2b496f9b5d35b134fb3fe28cd088336859bc4d8f7"}
Oct 30 17:15:01.263766 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.263745    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" event={"ID":"14ba928c-940f-4a37-af0d-0eef4245519e","Type":"ContainerStarted","Data":"212ff74aaa0d949899f363cc7f94f235ce05e41d5b923ac7be1404138e90a2f7"}
Oct 30 17:15:01.264966 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.264945    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" event={"ID":"a40ddc9f-a668-4958-81a5-cac8f457d495","Type":"ContainerStarted","Data":"f8ac7f1913cf214c711bd0b12b963711d346b20077436525dd286bd06bcd489f"}
Oct 30 17:15:01.266162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.266143    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" event={"ID":"ead4d5e1-6dea-496d-bdae-2b24341cdc13","Type":"ContainerStarted","Data":"4536d4acc2e03efb098f3e13c1f66eb2922f6aa46ddb523180a3756a253a1998"}
Oct 30 17:15:01.267365 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:01.267343    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" event={"ID":"04408a47-037c-41d2-95fa-f97ee17788eb","Type":"ContainerStarted","Data":"7fc02120f7d281d6acbceb4bc04d71e9eeb5d0faf0d9fae18bba4297e69f8273"}
Oct 30 17:15:02.295696 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.295669    4497 scope.go:122] "RemoveContainer" containerID="e7820b49cc8052a5aab71fdc85cf3d766b3671a03cf1b5780062b3c1bc73b851"
Oct 30 17:15:02.297226 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.297178    4497 generic.go:356] "Generic (PLEG): container finished" podID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerID="1af8d6a45cc95e3b2a7d16ae73eb6f3de7ea8d1247fc6dfca364dcf6edb5f9b3" exitCode=60
Oct 30 17:15:02.297593 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.297563    4497 scope.go:122] "RemoveContainer" containerID="1af8d6a45cc95e3b2a7d16ae73eb6f3de7ea8d1247fc6dfca364dcf6edb5f9b3"
Oct 30 17:15:02.299603 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.299442    4497 scope.go:122] "RemoveContainer" containerID="508c115e9cb89cd584c226a0c2d70ec27e70306afeccc9b00cbcd5131131b904"
Oct 30 17:15:02.301015 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.300917    4497 generic.go:356] "Generic (PLEG): container finished" podID="04408a47-037c-41d2-95fa-f97ee17788eb" containerID="7e9683fe2d48d7af93096ff988d99e48efcd71f5dcb5ccbc4d4de374fc5e5c1b" exitCode=7
Oct 30 17:15:02.301311 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.301294    4497 scope.go:122] "RemoveContainer" containerID="7e9683fe2d48d7af93096ff988d99e48efcd71f5dcb5ccbc4d4de374fc5e5c1b"
Oct 30 17:15:02.304215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.304176    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" event={"ID":"17877f0b-5c58-46f9-9e99-7c4afe23c8e3","Type":"ContainerStarted","Data":"fa43e800639dc0fd1f1888db0fd8179d9b1eca5e753b1ac38a2ee758b7408332"}
Oct 30 17:15:02.304299 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.304240    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" event={"ID":"14ba928c-940f-4a37-af0d-0eef4245519e","Type":"ContainerStarted","Data":"e7820b49cc8052a5aab71fdc85cf3d766b3671a03cf1b5780062b3c1bc73b851"}
Oct 30 17:15:02.304299 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.304262    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" event={"ID":"a40ddc9f-a668-4958-81a5-cac8f457d495","Type":"ContainerDied","Data":"1af8d6a45cc95e3b2a7d16ae73eb6f3de7ea8d1247fc6dfca364dcf6edb5f9b3"}
Oct 30 17:15:02.304299 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.304279    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" event={"ID":"ead4d5e1-6dea-496d-bdae-2b24341cdc13","Type":"ContainerStarted","Data":"508c115e9cb89cd584c226a0c2d70ec27e70306afeccc9b00cbcd5131131b904"}
Oct 30 17:15:02.304299 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.304296    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" event={"ID":"04408a47-037c-41d2-95fa-f97ee17788eb","Type":"ContainerDied","Data":"7e9683fe2d48d7af93096ff988d99e48efcd71f5dcb5ccbc4d4de374fc5e5c1b"}
Oct 30 17:15:02.334589 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:02.334538    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" podStartSLOduration=2.334519989 podStartE2EDuration="2.334519989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.329248213 +0000 UTC m=+3132.183126478" watchObservedRunningTime="2026-10-30 17:15:02.334519989 +0000 UTC m=+3132.188398255"
Oct 30 17:15:03.311223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.311155    4497 generic.go:356] "Generic (PLEG): container finished" podID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerID="fa43e800639dc0fd1f1888db0fd8179d9b1eca5e753b1ac38a2ee758b7408332" exitCode=60
Oct 30 17:15:03.311616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.311217    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" event={"ID":"17877f0b-5c58-46f9-9e99-7c4afe23c8e3","Type":"ContainerDied","Data":"fa43e800639dc0fd1f1888db0fd8179d9b1eca5e753b1ac38a2ee758b7408332"}
Oct 30 17:15:03.311616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.311475    4497 scope.go:122] "RemoveContainer" containerID="fa43e800639dc0fd1f1888db0fd8179d9b1eca5e753b1ac38a2ee758b7408332"
Oct 30 17:15:03.313506 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.313485    4497 generic.go:356] "Generic (PLEG): container finished" podID="14ba928c-940f-4a37-af0d-0eef4245519e" containerID="e7820b49cc8052a5aab71fdc85cf3d766b3671a03cf1b5780062b3c1bc73b851" exitCode=60
Oct 30 17:15:03.313626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.313570    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" event={"ID":"14ba928c-940f-4a37-af0d-0eef4245519e","Type":"ContainerDied","Data":"e7820b49cc8052a5aab71fdc85cf3d766b3671a03cf1b5780062b3c1bc73b851"}
Oct 30 17:15:03.313696 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.313623    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" event={"ID":"14ba928c-940f-4a37-af0d-0eef4245519e","Type":"ContainerStarted","Data":"7b07efabc1575a8fa2580e66178939b2081fa753fa003980ca59b7cac0e2595d"}
Oct 30 17:15:03.320447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.320410    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" event={"ID":"a40ddc9f-a668-4958-81a5-cac8f457d495","Type":"ContainerStarted","Data":"c82e3c52fe2031db07b3197baac5c7f9e6391b9f4a536ea71a070423d31e502e"}
Oct 30 17:15:03.322334 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.322315    4497 generic.go:356] "Generic (PLEG): container finished" podID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerID="508c115e9cb89cd584c226a0c2d70ec27e70306afeccc9b00cbcd5131131b904" exitCode=60
Oct 30 17:15:03.322447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.322399    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" event={"ID":"ead4d5e1-6dea-496d-bdae-2b24341cdc13","Type":"ContainerDied","Data":"508c115e9cb89cd584c226a0c2d70ec27e70306afeccc9b00cbcd5131131b904"}
Oct 30 17:15:03.322520 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.322450    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" event={"ID":"ead4d5e1-6dea-496d-bdae-2b24341cdc13","Type":"ContainerStarted","Data":"0c507b7d610dd59541b1ae2d50ae9e00dcea75665567aa0d1fab6f6d7456f8d9"}
Oct 30 17:15:03.324574 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.324549    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" event={"ID":"04408a47-037c-41d2-95fa-f97ee17788eb","Type":"ContainerStarted","Data":"b4b81f78c08eb76d52de3ecca56613eeb22680fb4ab39ecdff1a475f855206f4"}
Oct 30 17:15:03.385639 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.385590    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" podStartSLOduration=3.385575397 podStartE2EDuration="3.385575397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.379997026 +0000 UTC m=+3133.233875293" watchObservedRunningTime="2026-10-30 17:15:03.385575397 +0000 UTC m=+3133.239453662"
Oct 30 17:15:03.415887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.415840    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" podStartSLOduration=3.415825889 podStartE2EDuration="3.415825889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.413999505 +0000 UTC m=+3133.267877769" watchObservedRunningTime="2026-10-30 17:15:03.415825889 +0000 UTC m=+3133.269704154"
Oct 30 17:15:03.449834 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.449782    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" podStartSLOduration=3.449771156 podStartE2EDuration="3.449771156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.448415533 +0000 UTC m=+3133.302293798" watchObservedRunningTime="2026-10-30 17:15:03.449771156 +0000 UTC m=+3133.303649421"
Oct 30 17:15:03.483197 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:03.483132    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" podStartSLOduration=3.483115953 podStartE2EDuration="3.483115953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.482234196 +0000 UTC m=+3133.336112462" watchObservedRunningTime="2026-10-30 17:15:03.483115953 +0000 UTC m=+3133.336994224"
Oct 30 17:15:04.334401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.334364    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" event={"ID":"17877f0b-5c58-46f9-9e99-7c4afe23c8e3","Type":"ContainerStarted","Data":"68ae54a8b6cae587e6446045ed86eabff6ac9a657a1288dbd61a9d3bf87ce317"}
Oct 30 17:15:04.336320 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.336290    4497 generic.go:356] "Generic (PLEG): container finished" podID="14ba928c-940f-4a37-af0d-0eef4245519e" containerID="7b07efabc1575a8fa2580e66178939b2081fa753fa003980ca59b7cac0e2595d" exitCode=60
Oct 30 17:15:04.336441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.336370    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" event={"ID":"14ba928c-940f-4a37-af0d-0eef4245519e","Type":"ContainerDied","Data":"7b07efabc1575a8fa2580e66178939b2081fa753fa003980ca59b7cac0e2595d"}
Oct 30 17:15:04.336502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.336438    4497 scope.go:122] "RemoveContainer" containerID="e7820b49cc8052a5aab71fdc85cf3d766b3671a03cf1b5780062b3c1bc73b851"
Oct 30 17:15:04.336608 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.336592    4497 scope.go:122] "RemoveContainer" containerID="7b07efabc1575a8fa2580e66178939b2081fa753fa003980ca59b7cac0e2595d"
Oct 30 17:15:04.336913 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:04.336891    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7_odh-ai-gateway-infra(14ba928c-940f-4a37-af0d-0eef4245519e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" podUID="14ba928c-940f-4a37-af0d-0eef4245519e"
Oct 30 17:15:04.338282 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.338258    4497 generic.go:356] "Generic (PLEG): container finished" podID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerID="c82e3c52fe2031db07b3197baac5c7f9e6391b9f4a536ea71a070423d31e502e" exitCode=60
Oct 30 17:15:04.338403 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.338339    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" event={"ID":"a40ddc9f-a668-4958-81a5-cac8f457d495","Type":"ContainerDied","Data":"c82e3c52fe2031db07b3197baac5c7f9e6391b9f4a536ea71a070423d31e502e"}
Oct 30 17:15:04.338547 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.338532    4497 scope.go:122] "RemoveContainer" containerID="c82e3c52fe2031db07b3197baac5c7f9e6391b9f4a536ea71a070423d31e502e"
Oct 30 17:15:04.338853 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:04.338829    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn_odh-ai-gateway-infra(a40ddc9f-a668-4958-81a5-cac8f457d495)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495"
Oct 30 17:15:04.340222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.340201    4497 generic.go:356] "Generic (PLEG): container finished" podID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerID="0c507b7d610dd59541b1ae2d50ae9e00dcea75665567aa0d1fab6f6d7456f8d9" exitCode=60
Oct 30 17:15:04.340346 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.340284    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" event={"ID":"ead4d5e1-6dea-496d-bdae-2b24341cdc13","Type":"ContainerDied","Data":"0c507b7d610dd59541b1ae2d50ae9e00dcea75665567aa0d1fab6f6d7456f8d9"}
Oct 30 17:15:04.340516 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.340499    4497 scope.go:122] "RemoveContainer" containerID="0c507b7d610dd59541b1ae2d50ae9e00dcea75665567aa0d1fab6f6d7456f8d9"
Oct 30 17:15:04.340777 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:04.340755    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct_odh-ai-gateway-infra(ead4d5e1-6dea-496d-bdae-2b24341cdc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13"
Oct 30 17:15:04.342062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.342041    4497 generic.go:356] "Generic (PLEG): container finished" podID="04408a47-037c-41d2-95fa-f97ee17788eb" containerID="b4b81f78c08eb76d52de3ecca56613eeb22680fb4ab39ecdff1a475f855206f4" exitCode=60
Oct 30 17:15:04.342158 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.342121    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" event={"ID":"04408a47-037c-41d2-95fa-f97ee17788eb","Type":"ContainerDied","Data":"b4b81f78c08eb76d52de3ecca56613eeb22680fb4ab39ecdff1a475f855206f4"}
Oct 30 17:15:04.342426 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.342409    4497 scope.go:122] "RemoveContainer" containerID="b4b81f78c08eb76d52de3ecca56613eeb22680fb4ab39ecdff1a475f855206f4"
Oct 30 17:15:04.342743 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:04.342722    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79_odh-ai-gateway-infra(04408a47-037c-41d2-95fa-f97ee17788eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" podUID="04408a47-037c-41d2-95fa-f97ee17788eb"
Oct 30 17:15:04.351066 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.351049    4497 scope.go:122] "RemoveContainer" containerID="1af8d6a45cc95e3b2a7d16ae73eb6f3de7ea8d1247fc6dfca364dcf6edb5f9b3"
Oct 30 17:15:04.367150 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.367132    4497 scope.go:122] "RemoveContainer" containerID="508c115e9cb89cd584c226a0c2d70ec27e70306afeccc9b00cbcd5131131b904"
Oct 30 17:15:04.459470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:04.459448    4497 scope.go:122] "RemoveContainer" containerID="7e9683fe2d48d7af93096ff988d99e48efcd71f5dcb5ccbc4d4de374fc5e5c1b"
Oct 30 17:15:05.353066 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:05.353028    4497 generic.go:356] "Generic (PLEG): container finished" podID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerID="68ae54a8b6cae587e6446045ed86eabff6ac9a657a1288dbd61a9d3bf87ce317" exitCode=60
Oct 30 17:15:05.353584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:05.353110    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" event={"ID":"17877f0b-5c58-46f9-9e99-7c4afe23c8e3","Type":"ContainerDied","Data":"68ae54a8b6cae587e6446045ed86eabff6ac9a657a1288dbd61a9d3bf87ce317"}
Oct 30 17:15:05.353584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:05.353173    4497 scope.go:122] "RemoveContainer" containerID="fa43e800639dc0fd1f1888db0fd8179d9b1eca5e753b1ac38a2ee758b7408332"
Oct 30 17:15:05.353584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:05.353401    4497 scope.go:122] "RemoveContainer" containerID="68ae54a8b6cae587e6446045ed86eabff6ac9a657a1288dbd61a9d3bf87ce317"
Oct 30 17:15:05.353756 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:05.353699    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm_odh-ai-gateway-infra(17877f0b-5c58-46f9-9e99-7c4afe23c8e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3"
Oct 30 17:15:05.356328 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:05.356305    4497 scope.go:122] "RemoveContainer" containerID="7b07efabc1575a8fa2580e66178939b2081fa753fa003980ca59b7cac0e2595d"
Oct 30 17:15:05.356650 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:05.356626    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7_odh-ai-gateway-infra(14ba928c-940f-4a37-af0d-0eef4245519e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" podUID="14ba928c-940f-4a37-af0d-0eef4245519e"
Oct 30 17:15:05.358532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:05.358504    4497 scope.go:122] "RemoveContainer" containerID="c82e3c52fe2031db07b3197baac5c7f9e6391b9f4a536ea71a070423d31e502e"
Oct 30 17:15:05.358820 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:05.358800    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn_odh-ai-gateway-infra(a40ddc9f-a668-4958-81a5-cac8f457d495)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495"
Oct 30 17:15:05.360389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:05.360370    4497 scope.go:122] "RemoveContainer" containerID="0c507b7d610dd59541b1ae2d50ae9e00dcea75665567aa0d1fab6f6d7456f8d9"
Oct 30 17:15:05.360662 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:05.360641    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct_odh-ai-gateway-infra(ead4d5e1-6dea-496d-bdae-2b24341cdc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13"
Oct 30 17:15:05.362752 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:05.362734    4497 scope.go:122] "RemoveContainer" containerID="b4b81f78c08eb76d52de3ecca56613eeb22680fb4ab39ecdff1a475f855206f4"
Oct 30 17:15:05.363001 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:05.362979    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79_odh-ai-gateway-infra(04408a47-037c-41d2-95fa-f97ee17788eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" podUID="04408a47-037c-41d2-95fa-f97ee17788eb"
Oct 30 17:15:06.376845 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:06.376819    4497 scope.go:122] "RemoveContainer" containerID="68ae54a8b6cae587e6446045ed86eabff6ac9a657a1288dbd61a9d3bf87ce317"
Oct 30 17:15:06.377299 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:06.377242    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm_odh-ai-gateway-infra(17877f0b-5c58-46f9-9e99-7c4afe23c8e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3"
Oct 30 17:15:13.201225 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.201013    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"]
Oct 30 17:15:13.207508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.207478    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.223980 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.223952    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"]
Oct 30 17:15:13.228439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.228416    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:15:13.261863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.261824    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ql4\" (UniqueName: \"kubernetes.io/projected/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-kube-api-access-f9ql4\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.262021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.261882    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.262235 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.262213    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.364466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.364415    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.364653 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.364507    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ql4\" (UniqueName: \"kubernetes.io/projected/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-kube-api-access-f9ql4\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.364750 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.364709    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.365514 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.365479    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.366800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.366779    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.403034 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.402997    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ql4\" (UniqueName: \"kubernetes.io/projected/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-kube-api-access-f9ql4\") pod \"maas-api-revoke-sub-353da911-kh97t\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.523065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.523031    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:13.861656 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:13.861625    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"]
Oct 30 17:15:13.863287 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:13.863263    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7459f5d_7db1_4bea_a17f_4a6d61ef3495.slice/crio-247a54ee7a3210baba950ddecceee55b1c249bc623cf3630f6e7347de100edc0 WatchSource:0}: Error finding container 247a54ee7a3210baba950ddecceee55b1c249bc623cf3630f6e7347de100edc0: Status 404 returned error can't find the container with id 247a54ee7a3210baba950ddecceee55b1c249bc623cf3630f6e7347de100edc0
Oct 30 17:15:14.517155 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:14.517108    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t" event={"ID":"e7459f5d-7db1-4bea-a17f-4a6d61ef3495","Type":"ContainerStarted","Data":"2f68ed6189e629c49783453a897fdf2c47ddc053384510505a425abbb305638a"}
Oct 30 17:15:14.517155 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:14.517159    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t" event={"ID":"e7459f5d-7db1-4bea-a17f-4a6d61ef3495","Type":"ContainerStarted","Data":"247a54ee7a3210baba950ddecceee55b1c249bc623cf3630f6e7347de100edc0"}
Oct 30 17:15:14.566321 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:14.566272    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t" podStartSLOduration=1.566258059 podStartE2EDuration="1.566258059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:14.55256649 +0000 UTC m=+3144.406444755" watchObservedRunningTime="2026-10-30 17:15:14.566258059 +0000 UTC m=+3144.420136324"
Oct 30 17:15:15.526107 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:15.526076    4497 generic.go:356] "Generic (PLEG): container finished" podID="e7459f5d-7db1-4bea-a17f-4a6d61ef3495" containerID="2f68ed6189e629c49783453a897fdf2c47ddc053384510505a425abbb305638a" exitCode=0
Oct 30 17:15:15.526518 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:15.526162    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t" event={"ID":"e7459f5d-7db1-4bea-a17f-4a6d61ef3495","Type":"ContainerDied","Data":"2f68ed6189e629c49783453a897fdf2c47ddc053384510505a425abbb305638a"}
Oct 30 17:15:16.708882 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.708854    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:16.770578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.770544    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-openshift-ca\") pod \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") "
Oct 30 17:15:16.770744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.770590    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-kube-api-access-f9ql4\" (UniqueName: \"kubernetes.io/projected/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-kube-api-access-f9ql4\") pod \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") "
Oct 30 17:15:16.770821 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.770746    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-xks-ca\") pod \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\" (UID: \"e7459f5d-7db1-4bea-a17f-4a6d61ef3495\") "
Oct 30 17:15:16.770979 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.770951    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-openshift-ca" pod "e7459f5d-7db1-4bea-a17f-4a6d61ef3495" (UID: "e7459f5d-7db1-4bea-a17f-4a6d61ef3495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:16.771627 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.771600    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-openshift-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:16.773154 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.773128    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-kube-api-access-f9ql4" pod "e7459f5d-7db1-4bea-a17f-4a6d61ef3495" (UID: "e7459f5d-7db1-4bea-a17f-4a6d61ef3495"). InnerVolumeSpecName "kube-api-access-f9ql4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:16.774316 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.774290    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-xks-ca" pod "e7459f5d-7db1-4bea-a17f-4a6d61ef3495" (UID: "e7459f5d-7db1-4bea-a17f-4a6d61ef3495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:15:16.873421 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.873339    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-maas-api-xks-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:16.873421 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:16.873369    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ql4\" (UniqueName: \"kubernetes.io/projected/e7459f5d-7db1-4bea-a17f-4a6d61ef3495-kube-api-access-f9ql4\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:17.292663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:17.292630    4497 scope.go:122] "RemoveContainer" containerID="c82e3c52fe2031db07b3197baac5c7f9e6391b9f4a536ea71a070423d31e502e"
Oct 30 17:15:17.292851 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:17.292835    4497 scope.go:122] "RemoveContainer" containerID="7b07efabc1575a8fa2580e66178939b2081fa753fa003980ca59b7cac0e2595d"
Oct 30 17:15:17.293016 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:17.292991    4497 scope.go:122] "RemoveContainer" containerID="0c507b7d610dd59541b1ae2d50ae9e00dcea75665567aa0d1fab6f6d7456f8d9"
Oct 30 17:15:17.293864 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:17.293644    4497 scope.go:122] "RemoveContainer" containerID="68ae54a8b6cae587e6446045ed86eabff6ac9a657a1288dbd61a9d3bf87ce317"
Oct 30 17:15:17.294412 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:17.294033    4497 scope.go:122] "RemoveContainer" containerID="b4b81f78c08eb76d52de3ecca56613eeb22680fb4ab39ecdff1a475f855206f4"
Oct 30 17:15:17.547079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:17.546993    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"
Oct 30 17:15:17.547079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:17.547020    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t" event={"ID":"e7459f5d-7db1-4bea-a17f-4a6d61ef3495","Type":"ContainerDied","Data":"247a54ee7a3210baba950ddecceee55b1c249bc623cf3630f6e7347de100edc0"}
Oct 30 17:15:17.547363 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:17.547111    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="247a54ee7a3210baba950ddecceee55b1c249bc623cf3630f6e7347de100edc0"
Oct 30 17:15:18.560252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:18.560214    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" event={"ID":"17877f0b-5c58-46f9-9e99-7c4afe23c8e3","Type":"ContainerStarted","Data":"a9ddb890692d49480ecda25981da28792a7a57f8fc99f22797e343b592814286"}
Oct 30 17:15:18.562450 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:18.562415    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" event={"ID":"14ba928c-940f-4a37-af0d-0eef4245519e","Type":"ContainerStarted","Data":"353f3e1b24c904e9cd09496d55d33c84c7008a25e8509f5e366991f37abb30ca"}
Oct 30 17:15:18.564579 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:18.564549    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" event={"ID":"a40ddc9f-a668-4958-81a5-cac8f457d495","Type":"ContainerStarted","Data":"662c1cb07cd375376a0098685cd838e441061642b18b8634a7828ce0fff38bb7"}
Oct 30 17:15:18.566646 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:18.566624    4497 generic.go:356] "Generic (PLEG): container finished" podID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerID="89490e5784da5f344dc9ccfa333879a88747aa17ff006acfe163ba0c013b8613" exitCode=60
Oct 30 17:15:18.566763 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:18.566699    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" event={"ID":"ead4d5e1-6dea-496d-bdae-2b24341cdc13","Type":"ContainerDied","Data":"89490e5784da5f344dc9ccfa333879a88747aa17ff006acfe163ba0c013b8613"}
Oct 30 17:15:18.566763 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:18.566750    4497 scope.go:122] "RemoveContainer" containerID="0c507b7d610dd59541b1ae2d50ae9e00dcea75665567aa0d1fab6f6d7456f8d9"
Oct 30 17:15:18.567116 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:18.567100    4497 scope.go:122] "RemoveContainer" containerID="89490e5784da5f344dc9ccfa333879a88747aa17ff006acfe163ba0c013b8613"
Oct 30 17:15:18.567464 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:18.567442    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct_odh-ai-gateway-infra(ead4d5e1-6dea-496d-bdae-2b24341cdc13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13"
Oct 30 17:15:18.568804 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:18.568781    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" event={"ID":"04408a47-037c-41d2-95fa-f97ee17788eb","Type":"ContainerStarted","Data":"9bbc9ccd0e8ea4b9b3a8ca03261f7cac00e9d97f698307c9fc57ad25c7c38c5b"}
Oct 30 17:15:19.594252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.592106    4497 generic.go:356] "Generic (PLEG): container finished" podID="04408a47-037c-41d2-95fa-f97ee17788eb" containerID="9bbc9ccd0e8ea4b9b3a8ca03261f7cac00e9d97f698307c9fc57ad25c7c38c5b" exitCode=60
Oct 30 17:15:19.594252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.592478    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" event={"ID":"04408a47-037c-41d2-95fa-f97ee17788eb","Type":"ContainerDied","Data":"9bbc9ccd0e8ea4b9b3a8ca03261f7cac00e9d97f698307c9fc57ad25c7c38c5b"}
Oct 30 17:15:19.594252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.592550    4497 scope.go:122] "RemoveContainer" containerID="b4b81f78c08eb76d52de3ecca56613eeb22680fb4ab39ecdff1a475f855206f4"
Oct 30 17:15:19.594252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.593130    4497 scope.go:122] "RemoveContainer" containerID="9bbc9ccd0e8ea4b9b3a8ca03261f7cac00e9d97f698307c9fc57ad25c7c38c5b"
Oct 30 17:15:19.594252 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:19.593628    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79_odh-ai-gateway-infra(04408a47-037c-41d2-95fa-f97ee17788eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" podUID="04408a47-037c-41d2-95fa-f97ee17788eb"
Oct 30 17:15:19.609337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.609285    4497 generic.go:356] "Generic (PLEG): container finished" podID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerID="a9ddb890692d49480ecda25981da28792a7a57f8fc99f22797e343b592814286" exitCode=60
Oct 30 17:15:19.609499 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.609438    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" event={"ID":"17877f0b-5c58-46f9-9e99-7c4afe23c8e3","Type":"ContainerDied","Data":"a9ddb890692d49480ecda25981da28792a7a57f8fc99f22797e343b592814286"}
Oct 30 17:15:19.609997 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.609974    4497 scope.go:122] "RemoveContainer" containerID="a9ddb890692d49480ecda25981da28792a7a57f8fc99f22797e343b592814286"
Oct 30 17:15:19.610542 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:19.610515    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm_odh-ai-gateway-infra(17877f0b-5c58-46f9-9e99-7c4afe23c8e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3"
Oct 30 17:15:19.615026 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.614974    4497 generic.go:356] "Generic (PLEG): container finished" podID="14ba928c-940f-4a37-af0d-0eef4245519e" containerID="353f3e1b24c904e9cd09496d55d33c84c7008a25e8509f5e366991f37abb30ca" exitCode=60
Oct 30 17:15:19.615162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.615071    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" event={"ID":"14ba928c-940f-4a37-af0d-0eef4245519e","Type":"ContainerDied","Data":"353f3e1b24c904e9cd09496d55d33c84c7008a25e8509f5e366991f37abb30ca"}
Oct 30 17:15:19.615620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.615596    4497 scope.go:122] "RemoveContainer" containerID="353f3e1b24c904e9cd09496d55d33c84c7008a25e8509f5e366991f37abb30ca"
Oct 30 17:15:19.616132 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:19.616101    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7_odh-ai-gateway-infra(14ba928c-940f-4a37-af0d-0eef4245519e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" podUID="14ba928c-940f-4a37-af0d-0eef4245519e"
Oct 30 17:15:19.620439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.618932    4497 generic.go:356] "Generic (PLEG): container finished" podID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerID="662c1cb07cd375376a0098685cd838e441061642b18b8634a7828ce0fff38bb7" exitCode=60
Oct 30 17:15:19.620439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.619043    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" event={"ID":"a40ddc9f-a668-4958-81a5-cac8f457d495","Type":"ContainerDied","Data":"662c1cb07cd375376a0098685cd838e441061642b18b8634a7828ce0fff38bb7"}
Oct 30 17:15:19.620439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.619593    4497 scope.go:122] "RemoveContainer" containerID="662c1cb07cd375376a0098685cd838e441061642b18b8634a7828ce0fff38bb7"
Oct 30 17:15:19.620439 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:19.620016    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn_odh-ai-gateway-infra(a40ddc9f-a668-4958-81a5-cac8f457d495)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495"
Oct 30 17:15:19.625933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.625906    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"]
Oct 30 17:15:19.640458 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.640433    4497 scope.go:122] "RemoveContainer" containerID="68ae54a8b6cae587e6446045ed86eabff6ac9a657a1288dbd61a9d3bf87ce317"
Oct 30 17:15:19.654272 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.654243    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"]
Oct 30 17:15:19.657140 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.657116    4497 scope.go:122] "RemoveContainer" containerID="7b07efabc1575a8fa2580e66178939b2081fa753fa003980ca59b7cac0e2595d"
Oct 30 17:15:19.672120 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.672098    4497 scope.go:122] "RemoveContainer" containerID="c82e3c52fe2031db07b3197baac5c7f9e6391b9f4a536ea71a070423d31e502e"
Oct 30 17:15:19.697810 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.697776    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"]
Oct 30 17:15:19.739008 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.738970    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"]
Oct 30 17:15:19.774242 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.772651    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"]
Oct 30 17:15:19.981445 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:19.981412    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:20.041259 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.041226    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead4d5e1-6dea-496d-bdae-2b24341cdc13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead4d5e1-6dea-496d-bdae-2b24341cdc13-maas-api-service-ca\") pod \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\" (UID: \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\") "
Oct 30 17:15:20.041634 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.041610    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead4d5e1-6dea-496d-bdae-2b24341cdc13-kube-api-access-9hjbt\" (UniqueName: \"kubernetes.io/projected/ead4d5e1-6dea-496d-bdae-2b24341cdc13-kube-api-access-9hjbt\") pod \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\" (UID: \"ead4d5e1-6dea-496d-bdae-2b24341cdc13\") "
Oct 30 17:15:20.043873 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.043832    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead4d5e1-6dea-496d-bdae-2b24341cdc13-maas-api-service-ca" pod "ead4d5e1-6dea-496d-bdae-2b24341cdc13" (UID: "ead4d5e1-6dea-496d-bdae-2b24341cdc13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:20.048762 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.048732    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead4d5e1-6dea-496d-bdae-2b24341cdc13-kube-api-access-9hjbt" pod "ead4d5e1-6dea-496d-bdae-2b24341cdc13" (UID: "ead4d5e1-6dea-496d-bdae-2b24341cdc13"). InnerVolumeSpecName "kube-api-access-9hjbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:20.145180 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.145149    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead4d5e1-6dea-496d-bdae-2b24341cdc13-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:20.145180 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.145176    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hjbt\" (UniqueName: \"kubernetes.io/projected/ead4d5e1-6dea-496d-bdae-2b24341cdc13-kube-api-access-9hjbt\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:20.639883 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.639857    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"
Oct 30 17:15:20.640344 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.639892    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct" event={"ID":"ead4d5e1-6dea-496d-bdae-2b24341cdc13","Type":"ContainerDied","Data":"4536d4acc2e03efb098f3e13c1f66eb2922f6aa46ddb523180a3756a253a1998"}
Oct 30 17:15:20.640344 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.639989    4497 scope.go:122] "RemoveContainer" containerID="89490e5784da5f344dc9ccfa333879a88747aa17ff006acfe163ba0c013b8613"
Oct 30 17:15:20.644998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.643672    4497 generic.go:356] "Generic (PLEG): container finished" podID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerID="b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677" exitCode=0
Oct 30 17:15:20.644998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.643706    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" event={"ID":"f3ee375e-34e5-45d1-b01c-61ce79ef4c33","Type":"ContainerDied","Data":"b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677"}
Oct 30 17:15:20.644998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.644549    4497 scope.go:122] "RemoveContainer" containerID="b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677"
Oct 30 17:15:20.649583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.649001    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:15:20.649583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.649092    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:15:20.649583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.649141    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:15:20.844792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.844766    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:20.937849 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.937800    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"]
Oct 30 17:15:20.942707 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.942675    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889675-zhpct"]
Oct 30 17:15:20.955896 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.955868    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:20.964475 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.964449    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a40ddc9f-a668-4958-81a5-cac8f457d495-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40ddc9f-a668-4958-81a5-cac8f457d495-maas-api-service-ca\") pod \"a40ddc9f-a668-4958-81a5-cac8f457d495\" (UID: \"a40ddc9f-a668-4958-81a5-cac8f457d495\") "
Oct 30 17:15:20.964616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.964536    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40ddc9f-a668-4958-81a5-cac8f457d495-kube-api-access-48hp8\" (UniqueName: \"kubernetes.io/projected/a40ddc9f-a668-4958-81a5-cac8f457d495-kube-api-access-48hp8\") pod \"a40ddc9f-a668-4958-81a5-cac8f457d495\" (UID: \"a40ddc9f-a668-4958-81a5-cac8f457d495\") "
Oct 30 17:15:20.965381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.965356    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a40ddc9f-a668-4958-81a5-cac8f457d495-maas-api-service-ca" pod "a40ddc9f-a668-4958-81a5-cac8f457d495" (UID: "a40ddc9f-a668-4958-81a5-cac8f457d495"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:20.972399 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:20.972372    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40ddc9f-a668-4958-81a5-cac8f457d495-kube-api-access-48hp8" pod "a40ddc9f-a668-4958-81a5-cac8f457d495" (UID: "a40ddc9f-a668-4958-81a5-cac8f457d495"). InnerVolumeSpecName "kube-api-access-48hp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:21.058004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.057973    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:21.062153 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.062117    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:21.066789 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.066766    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ba928c-940f-4a37-af0d-0eef4245519e-kube-api-access-59zbb\" (UniqueName: \"kubernetes.io/projected/14ba928c-940f-4a37-af0d-0eef4245519e-kube-api-access-59zbb\") pod \"14ba928c-940f-4a37-af0d-0eef4245519e\" (UID: \"14ba928c-940f-4a37-af0d-0eef4245519e\") "
Oct 30 17:15:21.066917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.066904    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ba928c-940f-4a37-af0d-0eef4245519e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ba928c-940f-4a37-af0d-0eef4245519e-maas-api-service-ca\") pod \"14ba928c-940f-4a37-af0d-0eef4245519e\" (UID: \"14ba928c-940f-4a37-af0d-0eef4245519e\") "
Oct 30 17:15:21.067723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.067362    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ba928c-940f-4a37-af0d-0eef4245519e-maas-api-service-ca" pod "14ba928c-940f-4a37-af0d-0eef4245519e" (UID: "14ba928c-940f-4a37-af0d-0eef4245519e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:21.067723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.067663    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a40ddc9f-a668-4958-81a5-cac8f457d495-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.067723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.067687    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ba928c-940f-4a37-af0d-0eef4245519e-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.067723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.067701    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hp8\" (UniqueName: \"kubernetes.io/projected/a40ddc9f-a668-4958-81a5-cac8f457d495-kube-api-access-48hp8\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.069025 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.069001    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ba928c-940f-4a37-af0d-0eef4245519e-kube-api-access-59zbb" pod "14ba928c-940f-4a37-af0d-0eef4245519e" (UID: "14ba928c-940f-4a37-af0d-0eef4245519e"). InnerVolumeSpecName "kube-api-access-59zbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:21.169005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.168918    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04408a47-037c-41d2-95fa-f97ee17788eb-kube-api-access-mpdvg\" (UniqueName: \"kubernetes.io/projected/04408a47-037c-41d2-95fa-f97ee17788eb-kube-api-access-mpdvg\") pod \"04408a47-037c-41d2-95fa-f97ee17788eb\" (UID: \"04408a47-037c-41d2-95fa-f97ee17788eb\") "
Oct 30 17:15:21.169169 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.169053    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-kube-api-access-kfh89\" (UniqueName: \"kubernetes.io/projected/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-kube-api-access-kfh89\") pod \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\" (UID: \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\") "
Oct 30 17:15:21.169169 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.169092    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-maas-api-service-ca\") pod \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\" (UID: \"17877f0b-5c58-46f9-9e99-7c4afe23c8e3\") "
Oct 30 17:15:21.169365 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.169350    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04408a47-037c-41d2-95fa-f97ee17788eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04408a47-037c-41d2-95fa-f97ee17788eb-maas-api-service-ca\") pod \"04408a47-037c-41d2-95fa-f97ee17788eb\" (UID: \"04408a47-037c-41d2-95fa-f97ee17788eb\") "
Oct 30 17:15:21.170228 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.170177    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59zbb\" (UniqueName: \"kubernetes.io/projected/14ba928c-940f-4a37-af0d-0eef4245519e-kube-api-access-59zbb\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.170614 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.170587    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04408a47-037c-41d2-95fa-f97ee17788eb-maas-api-service-ca" pod "04408a47-037c-41d2-95fa-f97ee17788eb" (UID: "04408a47-037c-41d2-95fa-f97ee17788eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:21.170705 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.170594    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-maas-api-service-ca" pod "17877f0b-5c58-46f9-9e99-7c4afe23c8e3" (UID: "17877f0b-5c58-46f9-9e99-7c4afe23c8e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:21.172495 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.172459    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-kube-api-access-kfh89" pod "17877f0b-5c58-46f9-9e99-7c4afe23c8e3" (UID: "17877f0b-5c58-46f9-9e99-7c4afe23c8e3"). InnerVolumeSpecName "kube-api-access-kfh89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:21.173055 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.173030    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04408a47-037c-41d2-95fa-f97ee17788eb-kube-api-access-mpdvg" pod "04408a47-037c-41d2-95fa-f97ee17788eb" (UID: "04408a47-037c-41d2-95fa-f97ee17788eb"). InnerVolumeSpecName "kube-api-access-mpdvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:21.271792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.271758    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpdvg\" (UniqueName: \"kubernetes.io/projected/04408a47-037c-41d2-95fa-f97ee17788eb-kube-api-access-mpdvg\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.271792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.271787    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfh89\" (UniqueName: \"kubernetes.io/projected/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-kube-api-access-kfh89\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.271792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.271798    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17877f0b-5c58-46f9-9e99-7c4afe23c8e3-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.272025 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.271807    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04408a47-037c-41d2-95fa-f97ee17788eb-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:21.660934 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.660906    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"
Oct 30 17:15:21.661464 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.660939    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm" event={"ID":"17877f0b-5c58-46f9-9e99-7c4afe23c8e3","Type":"ContainerDied","Data":"591b7f610f46a3cb0ebd72e2b496f9b5d35b134fb3fe28cd088336859bc4d8f7"}
Oct 30 17:15:21.661464 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.661006    4497 scope.go:122] "RemoveContainer" containerID="a9ddb890692d49480ecda25981da28792a7a57f8fc99f22797e343b592814286"
Oct 30 17:15:21.662591 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.662531    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"
Oct 30 17:15:21.662591 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.662541    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7" event={"ID":"14ba928c-940f-4a37-af0d-0eef4245519e","Type":"ContainerDied","Data":"212ff74aaa0d949899f363cc7f94f235ce05e41d5b923ac7be1404138e90a2f7"}
Oct 30 17:15:21.664159 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.664137    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"
Oct 30 17:15:21.672309 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.664157    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn" event={"ID":"a40ddc9f-a668-4958-81a5-cac8f457d495","Type":"ContainerDied","Data":"f8ac7f1913cf214c711bd0b12b963711d346b20077436525dd286bd06bcd489f"}
Oct 30 17:15:21.703516 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.702942    4497 scope.go:122] "RemoveContainer" containerID="353f3e1b24c904e9cd09496d55d33c84c7008a25e8509f5e366991f37abb30ca"
Oct 30 17:15:21.705364 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.705322    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" event={"ID":"f3ee375e-34e5-45d1-b01c-61ce79ef4c33","Type":"ContainerStarted","Data":"405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4"}
Oct 30 17:15:21.707198 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.707164    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"
Oct 30 17:15:21.707299 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.707166    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79" event={"ID":"04408a47-037c-41d2-95fa-f97ee17788eb","Type":"ContainerDied","Data":"7fc02120f7d281d6acbceb4bc04d71e9eeb5d0faf0d9fae18bba4297e69f8273"}
Oct 30 17:15:21.714631 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.714612    4497 scope.go:122] "RemoveContainer" containerID="662c1cb07cd375376a0098685cd838e441061642b18b8634a7828ce0fff38bb7"
Oct 30 17:15:21.724602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.724581    4497 scope.go:122] "RemoveContainer" containerID="9bbc9ccd0e8ea4b9b3a8ca03261f7cac00e9d97f698307c9fc57ad25c7c38c5b"
Oct 30 17:15:21.759549 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.759521    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"]
Oct 30 17:15:21.763743 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.763710    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-799c76-29889675-pnwd7"]
Oct 30 17:15:21.830923 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.830882    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"]
Oct 30 17:15:21.842362 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.842333    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9afbb2-29889675-xhj79"]
Oct 30 17:15:21.897219 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.886404    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"]
Oct 30 17:15:21.903418 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.903389    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889675-7s5qn"]
Oct 30 17:15:21.929176 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:21.929101    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:15:21.931087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.931064    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"]
Oct 30 17:15:21.935394 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:21.935370    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889675-645mm"]
Oct 30 17:15:22.296009 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:22.295641    4497 patch_prober.go:29] interesting pod/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 17:15:22.296009 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:22.295716    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 17:15:22.301782 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:22.298037    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:15:22.312341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:22.312314    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" path="/var/lib/kubelet/pods/04408a47-037c-41d2-95fa-f97ee17788eb/volumes"
Oct 30 17:15:22.312871 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:22.312851    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" path="/var/lib/kubelet/pods/14ba928c-940f-4a37-af0d-0eef4245519e/volumes"
Oct 30 17:15:22.313509 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:22.313488    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" path="/var/lib/kubelet/pods/17877f0b-5c58-46f9-9e99-7c4afe23c8e3/volumes"
Oct 30 17:15:22.314663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:22.314641    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" path="/var/lib/kubelet/pods/a40ddc9f-a668-4958-81a5-cac8f457d495/volumes"
Oct 30 17:15:22.315280 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:22.315261    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" path="/var/lib/kubelet/pods/ead4d5e1-6dea-496d-bdae-2b24341cdc13/volumes"
Oct 30 17:15:23.297652 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:23.297615    4497 patch_prober.go:29] interesting pod/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 17:15:23.298224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:23.298156    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 17:15:24.299347 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:24.297499    4497 patch_prober.go:29] interesting pod/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 17:15:24.299347 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:24.297606    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 17:15:25.291214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:25.285284    4497 patch_prober.go:29] interesting pod/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 17:15:25.291214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:25.285355    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 17:15:26.295212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:26.284537    4497 patch_prober.go:29] interesting pod/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 17:15:26.295212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:26.284604    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 17:15:27.280065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:27.280034    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:15:27.280528 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:27.280516    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:15:27.281341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:27.281322    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:15:36.464057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.464015    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"]
Oct 30 17:15:36.465108 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465082    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:36.465108 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465107    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465133    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e7459f5d-7db1-4bea-a17f-4a6d61ef3495" containerName="revoke-keys"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465141    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7459f5d-7db1-4bea-a17f-4a6d61ef3495" containerName="revoke-keys"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465156    4497 cpu_manager.go:414] "containerMap: removing container" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465162    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465208    4497 cpu_manager.go:414] "containerMap: removing container" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465215    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465233    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465239    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465252    4497 cpu_manager.go:414] "containerMap: removing container" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465259    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465278    4497 cpu_manager.go:414] "containerMap: removing container" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465285    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465298    4497 cpu_manager.go:414] "containerMap: removing container" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465305    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465320    4497 cpu_manager.go:414] "containerMap: removing container" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465327    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465345    4497 cpu_manager.go:414] "containerMap: removing container" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465352    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465363    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:36.465359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465370    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465388    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465394    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465414    4497 cpu_manager.go:414] "containerMap: removing container" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465420    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465623    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7459f5d-7db1-4bea-a17f-4a6d61ef3495" containerName="revoke-keys"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465641    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465656    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465665    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465672    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ba928c-940f-4a37-af0d-0eef4245519e" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465681    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465690    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465696    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465703    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465712    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465723    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465731    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="04408a47-037c-41d2-95fa-f97ee17788eb" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465740    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:36.466952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.465749    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:36.471612 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.471590    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.489499 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.489473    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"]
Oct 30 17:15:36.494036 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.494000    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f94fb5\""
Oct 30 17:15:36.494036 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.494015    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"payload-processing-external-model-e2e--ac654823-dockercfg-cbc85\""
Oct 30 17:15:36.494217 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.494036    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"openshift-service-ca.crt\""
Oct 30 17:15:36.494217 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.494051    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"agc-cred-90e9f0cba9dc\""
Oct 30 17:15:36.496473 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.496457    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"kube-root-ca.crt\""
Oct 30 17:15:36.496554 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.496485    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"routing-overlay\""
Oct 30 17:15:36.595489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.595463    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.595650 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.595517    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.595708 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.595688    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.595775 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.595733    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns4tq\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-kube-api-access-ns4tq\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.697435 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.697407    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.697619 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.697455    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4tq\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-kube-api-access-ns4tq\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.697619 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.697584    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.697762 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.697642    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.698156 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.698129    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.698486 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.698460    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.700018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.700000    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.723975 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.723868    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4tq\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-kube-api-access-ns4tq\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:36.788205 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:36.787735    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:37.287574 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:37.286737    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15c8b3c_02de_411c_a891_de67bfa201af.slice/crio-865af49426ae7e87f328e589255489c5384e7be658839c1b51d7cbe021e5da46 WatchSource:0}: Error finding container 865af49426ae7e87f328e589255489c5384e7be658839c1b51d7cbe021e5da46: Status 404 returned error can't find the container with id 865af49426ae7e87f328e589255489c5384e7be658839c1b51d7cbe021e5da46
Oct 30 17:15:37.291127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.291100    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"]
Oct 30 17:15:37.457022 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.456980    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"]
Oct 30 17:15:37.457884 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.457861    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:37.457884 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.457883    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:37.458070 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.457969    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:37.458070 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.457978    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:37.458070 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.458006    4497 cpu_manager.go:414] "containerMap: removing container" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:37.458070 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.458015    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17877f0b-5c58-46f9-9e99-7c4afe23c8e3" containerName="cleanup"
Oct 30 17:15:37.458341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.458208    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40ddc9f-a668-4958-81a5-cac8f457d495" containerName="cleanup"
Oct 30 17:15:37.458341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.458241    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead4d5e1-6dea-496d-bdae-2b24341cdc13" containerName="cleanup"
Oct 30 17:15:37.462040 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.462019    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.484643 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.484624    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"]
Oct 30 17:15:37.498846 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.498815    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.498979 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.498959    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.499057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.499033    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgctl\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-kube-api-access-bgctl\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.499154 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.499136    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.600559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.600483    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.600559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.600549    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.600805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.600568    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgctl\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-kube-api-access-bgctl\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.600805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.600613    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.601435 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.601406    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.601535 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.601520    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.603035 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.603011    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.621609 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.621583    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgctl\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-kube-api-access-bgctl\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.776788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.776753    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:37.900701 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.900602    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c" event={"ID":"f15c8b3c-02de-411c-a891-de67bfa201af","Type":"ContainerStarted","Data":"84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c"}
Oct 30 17:15:37.900701 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.900681    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:37.900906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.900709    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c" event={"ID":"f15c8b3c-02de-411c-a891-de67bfa201af","Type":"ContainerStarted","Data":"865af49426ae7e87f328e589255489c5384e7be658839c1b51d7cbe021e5da46"}
Oct 30 17:15:37.948112 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:37.948055    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c" podStartSLOduration=1.9480335389999999 podStartE2EDuration="1.948033539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:37.942526146 +0000 UTC m=+3167.796404439" watchObservedRunningTime="2026-10-30 17:15:37.948033539 +0000 UTC m=+3167.801911801"
Oct 30 17:15:38.087815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:38.087783    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"]
Oct 30 17:15:38.090098 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:38.090070    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79735b65_564a_4fc1_9609_fcf2e00ce2c5.slice/crio-e822d79828f765982f858e1d17aa67ab41bca73d056c310b6714ec756c886739 WatchSource:0}: Error finding container e822d79828f765982f858e1d17aa67ab41bca73d056c310b6714ec756c886739: Status 404 returned error can't find the container with id e822d79828f765982f858e1d17aa67ab41bca73d056c310b6714ec756c886739
Oct 30 17:15:38.910504 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:38.910463    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d" event={"ID":"79735b65-564a-4fc1-9609-fcf2e00ce2c5","Type":"ContainerStarted","Data":"635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229"}
Oct 30 17:15:38.911003 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:38.910529    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d" event={"ID":"79735b65-564a-4fc1-9609-fcf2e00ce2c5","Type":"ContainerStarted","Data":"e822d79828f765982f858e1d17aa67ab41bca73d056c310b6714ec756c886739"}
Oct 30 17:15:38.911003 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:38.910673    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:38.948294 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:38.948253    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d" podStartSLOduration=1.9482412770000002 podStartE2EDuration="1.948241277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:38.946778096 +0000 UTC m=+3168.800656360" watchObservedRunningTime="2026-10-30 17:15:38.948241277 +0000 UTC m=+3168.802119541"
Oct 30 17:15:48.913899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:48.913869    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:15:49.925498 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:49.925468    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:15:50.001228 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:50.001176    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"]
Oct 30 17:15:51.918021 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:51.917984    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:15:55.001956 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.001905    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq" podUID="37ad0006-0371-4e7c-97ad-2f69c095e20a" containerName="payload-processing" containerID="cri-o://705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056" gracePeriod=25
Oct 30 17:15:55.308307 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.308285    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:15:55.375809 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.375780    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-provider-credentials\") pod \"37ad0006-0371-4e7c-97ad-2f69c095e20a\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") "
Oct 30 17:15:55.376015 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.375825    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-plugins-config-volume\") pod \"37ad0006-0371-4e7c-97ad-2f69c095e20a\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") "
Oct 30 17:15:55.376015 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.375903    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-routing-overlay-volume\") pod \"37ad0006-0371-4e7c-97ad-2f69c095e20a\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") "
Oct 30 17:15:55.376158 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.376100    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-kube-api-access-xgph6\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-kube-api-access-xgph6\") pod \"37ad0006-0371-4e7c-97ad-2f69c095e20a\" (UID: \"37ad0006-0371-4e7c-97ad-2f69c095e20a\") "
Oct 30 17:15:55.376325 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.376297    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-routing-overlay-volume" pod "37ad0006-0371-4e7c-97ad-2f69c095e20a" (UID: "37ad0006-0371-4e7c-97ad-2f69c095e20a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:55.376471 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.376369    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-plugins-config-volume" pod "37ad0006-0371-4e7c-97ad-2f69c095e20a" (UID: "37ad0006-0371-4e7c-97ad-2f69c095e20a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:15:55.376953 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.376929    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:55.377075 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.377061    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37ad0006-0371-4e7c-97ad-2f69c095e20a-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:55.378600 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.378575    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-kube-api-access-xgph6" pod "37ad0006-0371-4e7c-97ad-2f69c095e20a" (UID: "37ad0006-0371-4e7c-97ad-2f69c095e20a"). InnerVolumeSpecName "kube-api-access-xgph6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:55.378799 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.378773    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-provider-credentials" pod "37ad0006-0371-4e7c-97ad-2f69c095e20a" (UID: "37ad0006-0371-4e7c-97ad-2f69c095e20a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:15:55.478524 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.478489    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgph6\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-kube-api-access-xgph6\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:55.478524 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:55.478518    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37ad0006-0371-4e7c-97ad-2f69c095e20a-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:15:56.064360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.064320    4497 generic.go:356] "Generic (PLEG): container finished" podID="37ad0006-0371-4e7c-97ad-2f69c095e20a" containerID="705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056" exitCode=0
Oct 30 17:15:56.064909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.064377    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"
Oct 30 17:15:56.064909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.064441    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq" event={"ID":"37ad0006-0371-4e7c-97ad-2f69c095e20a","Type":"ContainerDied","Data":"705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056"}
Oct 30 17:15:56.064909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.064496    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq" event={"ID":"37ad0006-0371-4e7c-97ad-2f69c095e20a","Type":"ContainerDied","Data":"48935ecf1bce5cbe883434f259da2f2f9365c2f4e7328d0708d9f25eca8d501e"}
Oct 30 17:15:56.064909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.064519    4497 scope.go:122] "RemoveContainer" containerID="705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056"
Oct 30 17:15:56.076955 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.076934    4497 scope.go:122] "RemoveContainer" containerID="705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056"
Oct 30 17:15:56.077305 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:56.077282    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056\": container with ID starting with 705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056 not found: ID does not exist" containerID="705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056"
Oct 30 17:15:56.077391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.077312    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056"} err="failed to get container status \"705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056\": rpc error: code = NotFound desc = could not find container \"705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056\": container with ID starting with 705b9ab37e2fc584ba1809816c190b4102f3620d4beaf4642e8c90a95b619056 not found: ID does not exist"
Oct 30 17:15:56.110487 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.110459    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"]
Oct 30 17:15:56.119320 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.119298    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-54f42zmbq"]
Oct 30 17:15:56.127679 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:15:56.127650    4497 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ad0006_0371_4e7c_97ad_2f69c095e20a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ad0006_0371_4e7c_97ad_2f69c095e20a.slice/crio-48935ecf1bce5cbe883434f259da2f2f9365c2f4e7328d0708d9f25eca8d501e\": RecentStats: unable to find data in memory cache]"
Oct 30 17:15:56.302058 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:56.302021    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ad0006-0371-4e7c-97ad-2f69c095e20a" path="/var/lib/kubelet/pods/37ad0006-0371-4e7c-97ad-2f69c095e20a/volumes"
Oct 30 17:15:57.951855 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:57.951819    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"]
Oct 30 17:15:57.952893 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:57.952867    4497 cpu_manager.go:414] "containerMap: removing container" podUID="37ad0006-0371-4e7c-97ad-2f69c095e20a" containerName="payload-processing"
Oct 30 17:15:57.952893 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:57.952891    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ad0006-0371-4e7c-97ad-2f69c095e20a" containerName="payload-processing"
Oct 30 17:15:57.953084 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:57.953065    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ad0006-0371-4e7c-97ad-2f69c095e20a" containerName="payload-processing"
Oct 30 17:15:57.958389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:57.958367    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:57.975747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:57.975724    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"]
Oct 30 17:15:57.980391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:57.980370    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bbc36f\""
Oct 30 17:15:58.015649 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:58.015606    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm55p\" (UniqueName: \"kubernetes.io/projected/525852cc-f693-4705-b159-d66720176b20-kube-api-access-qm55p\") pod \"payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl\" (UID: \"525852cc-f693-4705-b159-d66720176b20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:58.015844 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:58.015747    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525852cc-f693-4705-b159-d66720176b20-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl\" (UID: \"525852cc-f693-4705-b159-d66720176b20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:58.132212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:58.129395    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525852cc-f693-4705-b159-d66720176b20-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl\" (UID: \"525852cc-f693-4705-b159-d66720176b20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:58.132212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:58.129812    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm55p\" (UniqueName: \"kubernetes.io/projected/525852cc-f693-4705-b159-d66720176b20-kube-api-access-qm55p\") pod \"payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl\" (UID: \"525852cc-f693-4705-b159-d66720176b20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:58.132212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:58.131108    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525852cc-f693-4705-b159-d66720176b20-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl\" (UID: \"525852cc-f693-4705-b159-d66720176b20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:58.196370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:58.196306    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm55p\" (UniqueName: \"kubernetes.io/projected/525852cc-f693-4705-b159-d66720176b20-kube-api-access-qm55p\") pod \"payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl\" (UID: \"525852cc-f693-4705-b159-d66720176b20\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:58.288532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:58.288486    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:58.637682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:58.637656    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"]
Oct 30 17:15:58.640139 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:15:58.640112    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525852cc_f693_4705_b159_d66720176b20.slice/crio-bdf908c0773a00b3fbdc531800b904e11c9f6049fd1405b89dff6e9f3bb20fd0 WatchSource:0}: Error finding container bdf908c0773a00b3fbdc531800b904e11c9f6049fd1405b89dff6e9f3bb20fd0: Status 404 returned error can't find the container with id bdf908c0773a00b3fbdc531800b904e11c9f6049fd1405b89dff6e9f3bb20fd0
Oct 30 17:15:59.093414 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:59.093379    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl" event={"ID":"525852cc-f693-4705-b159-d66720176b20","Type":"ContainerStarted","Data":"47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"}
Oct 30 17:15:59.093871 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:59.093430    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl" event={"ID":"525852cc-f693-4705-b159-d66720176b20","Type":"ContainerStarted","Data":"bdf908c0773a00b3fbdc531800b904e11c9f6049fd1405b89dff6e9f3bb20fd0"}
Oct 30 17:15:59.093871 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:59.093601    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:15:59.139470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:15:59.139419    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl" podStartSLOduration=2.139404194 podStartE2EDuration="2.139404194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:59.135128504 +0000 UTC m=+3188.989006770" watchObservedRunningTime="2026-10-30 17:15:59.139404194 +0000 UTC m=+3188.993282459"
Oct 30 17:16:00.251677 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:00.251646    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"]
Oct 30 17:16:01.162373 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.162336    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"]
Oct 30 17:16:01.342010 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.341978    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"]
Oct 30 17:16:01.347213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.347164    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.379893 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.379859    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9afbb2-openshift-default-dockercfg-vkv8d\""
Oct 30 17:16:01.388552 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.388529    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"]
Oct 30 17:16:01.495539 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.495468    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.495539 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.495502    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.495737 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.495595    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.495777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.495747    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b635ded-7d16-47e4-a0c6-a71d0a131533-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.496010 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.495984    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.496104 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.496087    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.496143 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.496123    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2zj\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-kube-api-access-pq2zj\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.496196 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.496171    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.496237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.496227    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.603887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.603852    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.604211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.604175    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.604353 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.604335    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2zj\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-kube-api-access-pq2zj\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.604470 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.604453    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.604584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.604569    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.604787 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.604772    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.604897 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.604884    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.605004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.604991    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.605206 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.605171    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b635ded-7d16-47e4-a0c6-a71d0a131533-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.606224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.606179    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b635ded-7d16-47e4-a0c6-a71d0a131533-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.607443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.607420    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.608040 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.608019    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.619333 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.618468    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.619333 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.618726    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.619333 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.618953    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.658924 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.658847    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.660277 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.660244    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2zj\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-kube-api-access-pq2zj\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.673851 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.671726    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:01.964403 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:01.964374    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:02.314991 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:02.314963    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"]
Oct 30 17:16:02.317170 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:16:02.317138    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b635ded_7d16_47e4_a0c6_a71d0a131533.slice/crio-204ae1af1873797fd652e348388e006a322e97b2a404167e5e6bace113f5cd6c WatchSource:0}: Error finding container 204ae1af1873797fd652e348388e006a322e97b2a404167e5e6bace113f5cd6c: Status 404 returned error can't find the container with id 204ae1af1873797fd652e348388e006a322e97b2a404167e5e6bace113f5cd6c
Oct 30 17:16:02.320379 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:02.320345    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:16:02.320489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:02.320412    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:16:02.320489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:02.320441    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:16:03.155075 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:03.155037    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" event={"ID":"6b635ded-7d16-47e4-a0c6-a71d0a131533","Type":"ContainerStarted","Data":"6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5"}
Oct 30 17:16:03.155529 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:03.155099    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" event={"ID":"6b635ded-7d16-47e4-a0c6-a71d0a131533","Type":"ContainerStarted","Data":"204ae1af1873797fd652e348388e006a322e97b2a404167e5e6bace113f5cd6c"}
Oct 30 17:16:03.222906 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:03.222856    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" podStartSLOduration=2.222842703 podStartE2EDuration="2.222842703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:03.213057701 +0000 UTC m=+3193.066935966" watchObservedRunningTime="2026-10-30 17:16:03.222842703 +0000 UTC m=+3193.076720967"
Oct 30 17:16:03.965126 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:03.965092    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:03.966438 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:03.966417    4497 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 17:16:03.966503 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:03.966462    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 17:16:04.964958 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:04.964920    4497 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 17:16:04.965396 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:04.964983    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 17:16:05.252956 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.252869    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c" podUID="f15c8b3c-02de-411c-a891-de67bfa201af" containerName="payload-processing" containerID="cri-o://84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c" gracePeriod=25
Oct 30 17:16:05.610917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.610891    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:16:05.703941 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.703808    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-routing-overlay-volume\") pod \"f15c8b3c-02de-411c-a891-de67bfa201af\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") "
Oct 30 17:16:05.703941 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.703900    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-plugins-config-volume\") pod \"f15c8b3c-02de-411c-a891-de67bfa201af\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") "
Oct 30 17:16:05.707521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.704297    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-provider-credentials\") pod \"f15c8b3c-02de-411c-a891-de67bfa201af\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") "
Oct 30 17:16:05.707521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.704332    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-kube-api-access-ns4tq\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-kube-api-access-ns4tq\") pod \"f15c8b3c-02de-411c-a891-de67bfa201af\" (UID: \"f15c8b3c-02de-411c-a891-de67bfa201af\") "
Oct 30 17:16:05.707521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.704522    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-routing-overlay-volume" pod "f15c8b3c-02de-411c-a891-de67bfa201af" (UID: "f15c8b3c-02de-411c-a891-de67bfa201af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:05.707521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.704989    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-plugins-config-volume" pod "f15c8b3c-02de-411c-a891-de67bfa201af" (UID: "f15c8b3c-02de-411c-a891-de67bfa201af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:05.707521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.705083    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:05.707521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.705105    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15c8b3c-02de-411c-a891-de67bfa201af-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:05.707925 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.707896    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-provider-credentials" pod "f15c8b3c-02de-411c-a891-de67bfa201af" (UID: "f15c8b3c-02de-411c-a891-de67bfa201af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:05.708072 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.707941    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-kube-api-access-ns4tq" pod "f15c8b3c-02de-411c-a891-de67bfa201af" (UID: "f15c8b3c-02de-411c-a891-de67bfa201af"). InnerVolumeSpecName "kube-api-access-ns4tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:05.807268 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.807128    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:05.807268 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.807168    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns4tq\" (UniqueName: \"kubernetes.io/projected/f15c8b3c-02de-411c-a891-de67bfa201af-kube-api-access-ns4tq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:05.965587 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.965554    4497 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 17:16:05.965990 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:05.965608    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 17:16:06.163113 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.163006    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d" podUID="79735b65-564a-4fc1-9609-fcf2e00ce2c5" containerName="payload-processing" containerID="cri-o://635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229" gracePeriod=25
Oct 30 17:16:06.208743 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.208708    4497 generic.go:356] "Generic (PLEG): container finished" podID="f15c8b3c-02de-411c-a891-de67bfa201af" containerID="84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c" exitCode=0
Oct 30 17:16:06.208890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.208752    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"
Oct 30 17:16:06.208890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.208772    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c" event={"ID":"f15c8b3c-02de-411c-a891-de67bfa201af","Type":"ContainerDied","Data":"84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c"}
Oct 30 17:16:06.208890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.208825    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c" event={"ID":"f15c8b3c-02de-411c-a891-de67bfa201af","Type":"ContainerDied","Data":"865af49426ae7e87f328e589255489c5384e7be658839c1b51d7cbe021e5da46"}
Oct 30 17:16:06.208890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.208854    4497 scope.go:122] "RemoveContainer" containerID="84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c"
Oct 30 17:16:06.310697 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.310663    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"]
Oct 30 17:16:06.311329 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.311111    4497 scope.go:122] "RemoveContainer" containerID="84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c"
Oct 30 17:16:06.311329 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.311243    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649cr96c"]
Oct 30 17:16:06.311521 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:16:06.311502    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c\": container with ID starting with 84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c not found: ID does not exist" containerID="84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c"
Oct 30 17:16:06.311561 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.311534    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c"} err="failed to get container status \"84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c\": rpc error: code = NotFound desc = could not find container \"84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c\": container with ID starting with 84051c2a8e02555bd7d35cd75c9932eedd280e8893e1134d9659fd0c7a4b150c not found: ID does not exist"
Oct 30 17:16:06.543247 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.543223    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:16:06.720276 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.720246    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-plugins-config-volume\") pod \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") "
Oct 30 17:16:06.720440 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.720315    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-routing-overlay-volume\") pod \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") "
Oct 30 17:16:06.720603 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.720578    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-provider-credentials\") pod \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") "
Oct 30 17:16:06.720776 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.720615    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-kube-api-access-bgctl\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-kube-api-access-bgctl\") pod \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\" (UID: \"79735b65-564a-4fc1-9609-fcf2e00ce2c5\") "
Oct 30 17:16:06.720776 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.720725    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-plugins-config-volume" pod "79735b65-564a-4fc1-9609-fcf2e00ce2c5" (UID: "79735b65-564a-4fc1-9609-fcf2e00ce2c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:06.720776 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.720737    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-routing-overlay-volume" pod "79735b65-564a-4fc1-9609-fcf2e00ce2c5" (UID: "79735b65-564a-4fc1-9609-fcf2e00ce2c5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:06.721234 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.721214    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:06.721378 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.721363    4497 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79735b65-564a-4fc1-9609-fcf2e00ce2c5-routing-overlay-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:06.722813 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.722791    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-provider-credentials" pod "79735b65-564a-4fc1-9609-fcf2e00ce2c5" (UID: "79735b65-564a-4fc1-9609-fcf2e00ce2c5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:06.722937 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.722914    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-kube-api-access-bgctl" pod "79735b65-564a-4fc1-9609-fcf2e00ce2c5" (UID: "79735b65-564a-4fc1-9609-fcf2e00ce2c5"). InnerVolumeSpecName "kube-api-access-bgctl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:06.823447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.823362    4497 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-provider-credentials\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:06.823447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.823390    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgctl\" (UniqueName: \"kubernetes.io/projected/79735b65-564a-4fc1-9609-fcf2e00ce2c5-kube-api-access-bgctl\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:06.964794 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.964764    4497 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 17:16:06.964970 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:06.964809    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 17:16:07.218252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.218164    4497 generic.go:356] "Generic (PLEG): container finished" podID="79735b65-564a-4fc1-9609-fcf2e00ce2c5" containerID="635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229" exitCode=0
Oct 30 17:16:07.218252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.218231    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"
Oct 30 17:16:07.218840 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.218231    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d" event={"ID":"79735b65-564a-4fc1-9609-fcf2e00ce2c5","Type":"ContainerDied","Data":"635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229"}
Oct 30 17:16:07.218840 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.218348    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d" event={"ID":"79735b65-564a-4fc1-9609-fcf2e00ce2c5","Type":"ContainerDied","Data":"e822d79828f765982f858e1d17aa67ab41bca73d056c310b6714ec756c886739"}
Oct 30 17:16:07.218840 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.218379    4497 scope.go:122] "RemoveContainer" containerID="635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229"
Oct 30 17:16:07.232093 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.232075    4497 scope.go:122] "RemoveContainer" containerID="635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229"
Oct 30 17:16:07.232378 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:16:07.232361    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229\": container with ID starting with 635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229 not found: ID does not exist" containerID="635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229"
Oct 30 17:16:07.232464 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.232384    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229"} err="failed to get container status \"635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229\": rpc error: code = NotFound desc = could not find container \"635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229\": container with ID starting with 635aa3602630632f35629fbe0e7b5e829574a3345a5ed1402e0e546ca982d229 not found: ID does not exist"
Oct 30 17:16:07.294211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.292738    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"]
Oct 30 17:16:07.301964 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.301933    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbd6dg2d"]
Oct 30 17:16:07.965384 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.965351    4497 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 17:16:07.965570 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:07.965404    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 17:16:08.305049 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:08.305015    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79735b65-564a-4fc1-9609-fcf2e00ce2c5" path="/var/lib/kubelet/pods/79735b65-564a-4fc1-9609-fcf2e00ce2c5/volumes"
Oct 30 17:16:08.305748 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:08.305725    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15c8b3c-02de-411c-a891-de67bfa201af" path="/var/lib/kubelet/pods/f15c8b3c-02de-411c-a891-de67bfa201af/volumes"
Oct 30 17:16:08.970707 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:08.970675    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:08.971162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:08.971140    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:08.971948 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:08.971923    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:16:10.130121 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:10.130094    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:16:13.704255 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:13.704217    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"]
Oct 30 17:16:13.704883 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:13.704545    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy" containerID="cri-o://772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68" gracePeriod=30
Oct 30 17:16:14.988298 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:14.984896    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"]
Oct 30 17:16:15.050209 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.048328    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f15c8b3c-02de-411c-a891-de67bfa201af" containerName="payload-processing"
Oct 30 17:16:15.050209 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.048360    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15c8b3c-02de-411c-a891-de67bfa201af" containerName="payload-processing"
Oct 30 17:16:15.050209 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.048417    4497 cpu_manager.go:414] "containerMap: removing container" podUID="79735b65-564a-4fc1-9609-fcf2e00ce2c5" containerName="payload-processing"
Oct 30 17:16:15.050209 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.048426    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79735b65-564a-4fc1-9609-fcf2e00ce2c5" containerName="payload-processing"
Oct 30 17:16:15.050209 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.048662    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15c8b3c-02de-411c-a891-de67bfa201af" containerName="payload-processing"
Oct 30 17:16:15.050209 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.048682    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="79735b65-564a-4fc1-9609-fcf2e00ce2c5" containerName="payload-processing"
Oct 30 17:16:15.102221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.099426    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"]
Oct 30 17:16:15.102221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.099578    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.183210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.177673    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24acb2-openshift-default-dockercfg-nrq8m\""
Oct 30 17:16:15.225969 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.225921    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-envoy\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.226283 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.226256    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-data\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.226410 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.226294    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-istio-token\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.226480 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.226420    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.226480 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.226465    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3de812-692e-405a-953c-cc84de94107a-istio-podinfo\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.226611 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.226501    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3de812-692e-405a-953c-cc84de94107a-istiod-ca-cert\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.226611 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.226569    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jjlg\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-kube-api-access-6jjlg\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.226728 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.226620    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-certs\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.226728 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.226698    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-credential-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328329    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328373    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3de812-692e-405a-953c-cc84de94107a-istio-podinfo\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328397    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3de812-692e-405a-953c-cc84de94107a-istiod-ca-cert\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328439    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jjlg\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-kube-api-access-6jjlg\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328469    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-certs\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328533    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-credential-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328616    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-envoy\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328759    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-data\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328786    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-istio-token\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328785    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.328976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.328837    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-certs\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.329229 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.329055    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-credential-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.329288 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.329228    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3de812-692e-405a-953c-cc84de94107a-istiod-ca-cert\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.329575 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.329313    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-data\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.330952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.330924    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-envoy\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.331086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.331074    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3de812-692e-405a-953c-cc84de94107a-istio-podinfo\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.350807 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.350785    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-istio-token\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.350936 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.350843    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jjlg\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-kube-api-access-6jjlg\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.497089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.497053    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:15.829764 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.829733    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"]
Oct 30 17:16:15.831605 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:16:15.831576    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3de812_692e_405a_953c_cc84de94107a.slice/crio-ef3312599949a2e6fabc7085189620a0b78b0d38f3fbfd8202b96e3247fd0ce6 WatchSource:0}: Error finding container ef3312599949a2e6fabc7085189620a0b78b0d38f3fbfd8202b96e3247fd0ce6: Status 404 returned error can't find the container with id ef3312599949a2e6fabc7085189620a0b78b0d38f3fbfd8202b96e3247fd0ce6
Oct 30 17:16:15.834858 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.834823    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:16:15.834934 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.834892    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:16:15.834934 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:15.834920    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:16:16.152258 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.147231    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"]
Oct 30 17:16:16.187050 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.185777    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:16.226555 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.217937    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d80ebb\""
Oct 30 17:16:16.226555 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.218108    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d80ebb-dockercfg-2zpmm\""
Oct 30 17:16:16.226555 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.221954    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"]
Oct 30 17:16:16.317770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.317733    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" event={"ID":"7e3de812-692e-405a-953c-cc84de94107a","Type":"ContainerStarted","Data":"9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d"}
Oct 30 17:16:16.317971 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.317795    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" event={"ID":"7e3de812-692e-405a-953c-cc84de94107a","Type":"ContainerStarted","Data":"ef3312599949a2e6fabc7085189620a0b78b0d38f3fbfd8202b96e3247fd0ce6"}
Oct 30 17:16:16.364200 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.362856    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podStartSLOduration=2.362838893 podStartE2EDuration="2.362838893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:16.358671133 +0000 UTC m=+3206.212549399" watchObservedRunningTime="2026-10-30 17:16:16.362838893 +0000 UTC m=+3206.216717159"
Oct 30 17:16:16.387574 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.387543    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c48g\" (UniqueName: \"kubernetes.io/projected/2f7bff6b-8665-4536-8e4a-796474a61f05-kube-api-access-2c48g\") pod \"payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9\" (UID: \"2f7bff6b-8665-4536-8e4a-796474a61f05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:16.387728 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.387700    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7bff6b-8665-4536-8e4a-796474a61f05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9\" (UID: \"2f7bff6b-8665-4536-8e4a-796474a61f05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:16.489746 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.489659    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c48g\" (UniqueName: \"kubernetes.io/projected/2f7bff6b-8665-4536-8e4a-796474a61f05-kube-api-access-2c48g\") pod \"payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9\" (UID: \"2f7bff6b-8665-4536-8e4a-796474a61f05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:16.489949 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.489820    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7bff6b-8665-4536-8e4a-796474a61f05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9\" (UID: \"2f7bff6b-8665-4536-8e4a-796474a61f05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:16.490607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.490586    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7bff6b-8665-4536-8e4a-796474a61f05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9\" (UID: \"2f7bff6b-8665-4536-8e4a-796474a61f05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:16.497805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.497770    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:16.499077 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.499050    4497 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 17:16:16.499169 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.499105    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 17:16:16.511647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.511624    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c48g\" (UniqueName: \"kubernetes.io/projected/2f7bff6b-8665-4536-8e4a-796474a61f05-kube-api-access-2c48g\") pod \"payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9\" (UID: \"2f7bff6b-8665-4536-8e4a-796474a61f05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:16.597149 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.597115    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:16.932971 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:16.932947    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"]
Oct 30 17:16:16.935557 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:16:16.935529    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7bff6b_8665_4536_8e4a_796474a61f05.slice/crio-622231f285716b11ed17c577d16bf3ebfc130143acbaec4629ae10d0c42bdcf7 WatchSource:0}: Error finding container 622231f285716b11ed17c577d16bf3ebfc130143acbaec4629ae10d0c42bdcf7: Status 404 returned error can't find the container with id 622231f285716b11ed17c577d16bf3ebfc130143acbaec4629ae10d0c42bdcf7
Oct 30 17:16:17.328428 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:17.328385    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9" event={"ID":"2f7bff6b-8665-4536-8e4a-796474a61f05","Type":"ContainerStarted","Data":"0d9e06d6a59559abc4eae3f12708a213c125d71b06165955e9afca0930f2289f"}
Oct 30 17:16:17.328890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:17.328457    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9" event={"ID":"2f7bff6b-8665-4536-8e4a-796474a61f05","Type":"ContainerStarted","Data":"622231f285716b11ed17c577d16bf3ebfc130143acbaec4629ae10d0c42bdcf7"}
Oct 30 17:16:17.328890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:17.328490    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:17.368763 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:17.368474    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9" podStartSLOduration=2.368457079 podStartE2EDuration="2.368457079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:17.366224225 +0000 UTC m=+3207.220102489" watchObservedRunningTime="2026-10-30 17:16:17.368457079 +0000 UTC m=+3207.222335344"
Oct 30 17:16:17.499237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:17.498635    4497 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 17:16:17.499237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:17.498716    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 17:16:18.497972 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:18.497934    4497 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 17:16:18.498439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:18.497990    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 17:16:19.038464 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.038437    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:16:19.203964 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.203872    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1f07ab-72d9-414a-abd2-d379e71a331f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f07ab-72d9-414a-abd2-d379e71a331f-istiod-ca-cert\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.204318 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.204294    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-data\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.204466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.204450    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-kube-api-access-5m29g\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-kube-api-access-5m29g\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.204635 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.204616    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-podinfo\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.204804 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.204787    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-credential-socket\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.205141 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.205119    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-certs\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.205559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.205540    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-token\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.205711 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.205694    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-socket\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.205863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.205847    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-envoy\") pod \"ec1f07ab-72d9-414a-abd2-d379e71a331f\" (UID: \"ec1f07ab-72d9-414a-abd2-d379e71a331f\") "
Oct 30 17:16:19.209744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.206109    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1f07ab-72d9-414a-abd2-d379e71a331f-istiod-ca-cert" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:19.209744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.206539    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-credential-socket" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:19.209744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.207291    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-socket" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:19.209744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.207737    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-data" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:19.209744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.207869    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-certs" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:19.209744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.209598    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-kube-api-access-5m29g" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "kube-api-access-5m29g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:19.210393 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.210345    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-token" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:19.212065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.211540    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec1f07ab-72d9-414a-abd2-d379e71a331f-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.212244 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.212175    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.212369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.212353    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m29g\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-kube-api-access-5m29g\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.212472 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.212455    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.212818 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.212802    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.213237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.213161    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-podinfo" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:16:19.213752 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.213721    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.213956 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.213940    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.214791 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.214766    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-envoy" pod "ec1f07ab-72d9-414a-abd2-d379e71a331f" (UID: "ec1f07ab-72d9-414a-abd2-d379e71a331f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:19.315891 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.315858    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.315891 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.315890    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec1f07ab-72d9-414a-abd2-d379e71a331f-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:19.350853 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.350821    4497 generic.go:356] "Generic (PLEG): container finished" podID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerID="772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68" exitCode=0
Oct 30 17:16:19.351006 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.350870    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"
Oct 30 17:16:19.351006 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.350903    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" event={"ID":"ec1f07ab-72d9-414a-abd2-d379e71a331f","Type":"ContainerDied","Data":"772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68"}
Oct 30 17:16:19.351006 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.350967    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj" event={"ID":"ec1f07ab-72d9-414a-abd2-d379e71a331f","Type":"ContainerDied","Data":"ae095aa4929f9e434576cc13796b7f32f06f717544508d789192473801cdf4ab"}
Oct 30 17:16:19.351006 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.350996    4497 scope.go:122] "RemoveContainer" containerID="772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68"
Oct 30 17:16:19.366073 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.366048    4497 scope.go:122] "RemoveContainer" containerID="772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68"
Oct 30 17:16:19.366459 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:16:19.366437    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68\": container with ID starting with 772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68 not found: ID does not exist" containerID="772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68"
Oct 30 17:16:19.366552 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.366468    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68"} err="failed to get container status \"772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68\": rpc error: code = NotFound desc = could not find container \"772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68\": container with ID starting with 772bb20abb9734a2b4b81fe0e16a40ed482f4576a4f9035be271c64f03ef2e68 not found: ID does not exist"
Oct 30 17:16:19.403386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.403355    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"]
Oct 30 17:16:19.415007 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.414968    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-lzxmj"]
Oct 30 17:16:19.430466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.430436    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"]
Oct 30 17:16:19.430670 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.430646    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy" containerID="cri-o://d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292" gracePeriod=30
Oct 30 17:16:19.501399 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.501313    4497 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 17:16:19.501399 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:19.501378    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 17:16:20.307319 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.307282    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" path="/var/lib/kubelet/pods/ec1f07ab-72d9-414a-abd2-d379e71a331f/volumes"
Oct 30 17:16:20.500211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.499600    4497 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 17:16:20.500211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.499660    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 17:16:20.773240 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.773208    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"]
Oct 30 17:16:20.799584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.799524    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy"
Oct 30 17:16:20.799584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.799553    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy"
Oct 30 17:16:20.800489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.799860    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1f07ab-72d9-414a-abd2-d379e71a331f" containerName="istio-proxy"
Oct 30 17:16:20.805972 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.805946    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"]
Oct 30 17:16:20.806068 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.806046    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:20.822644 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.820370    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-85eb87\""
Oct 30 17:16:20.854562 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.829523    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-85eb87-dockercfg-c8wzt\""
Oct 30 17:16:20.893212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.892405    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j\" (UID: \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:20.893212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.892683    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9pjl\" (UniqueName: \"kubernetes.io/projected/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-kube-api-access-q9pjl\") pod \"payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j\" (UID: \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:20.994437 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.994396    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j\" (UID: \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:20.994668 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:20.994649    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pjl\" (UniqueName: \"kubernetes.io/projected/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-kube-api-access-q9pjl\") pod \"payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j\" (UID: \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:21.018238 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.012238    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j\" (UID: \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:21.083266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.074047    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"]
Oct 30 17:16:21.169359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.169211    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"]
Oct 30 17:16:21.169359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.169353    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:21.196210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.192417    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pjl\" (UniqueName: \"kubernetes.io/projected/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-kube-api-access-q9pjl\") pod \"payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j\" (UID: \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:21.329096 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.329060    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrzx8\" (UniqueName: \"kubernetes.io/projected/066e972e-b5f4-4778-985e-e4a3174f4bfb-kube-api-access-nrzx8\") pod \"payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47\" (UID: \"066e972e-b5f4-4778-985e-e4a3174f4bfb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:21.329345 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.329325    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066e972e-b5f4-4778-985e-e4a3174f4bfb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47\" (UID: \"066e972e-b5f4-4778-985e-e4a3174f4bfb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:21.431175 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.431082    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrzx8\" (UniqueName: \"kubernetes.io/projected/066e972e-b5f4-4778-985e-e4a3174f4bfb-kube-api-access-nrzx8\") pod \"payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47\" (UID: \"066e972e-b5f4-4778-985e-e4a3174f4bfb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:21.431463 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.431440    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066e972e-b5f4-4778-985e-e4a3174f4bfb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47\" (UID: \"066e972e-b5f4-4778-985e-e4a3174f4bfb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:21.432123 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.432094    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066e972e-b5f4-4778-985e-e4a3174f4bfb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47\" (UID: \"066e972e-b5f4-4778-985e-e4a3174f4bfb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:21.434430 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.434413    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:21.452413 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.452385    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrzx8\" (UniqueName: \"kubernetes.io/projected/066e972e-b5f4-4778-985e-e4a3174f4bfb-kube-api-access-nrzx8\") pod \"payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47\" (UID: \"066e972e-b5f4-4778-985e-e4a3174f4bfb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:21.497566 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.497529    4497 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 17:16:21.497871 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.497825    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 17:16:21.527915 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.524675    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:21.933827 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:16:21.933783    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:16:21.997567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:21.987661    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"]
Oct 30 17:16:22.137564 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.137534    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"]
Oct 30 17:16:22.139564 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:16:22.139537    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066e972e_b5f4_4778_985e_e4a3174f4bfb.slice/crio-7a0fc1891b3d07bc445bc5ed4ad9d97e83f78045f398dda6794a27f561840aa1 WatchSource:0}: Error finding container 7a0fc1891b3d07bc445bc5ed4ad9d97e83f78045f398dda6794a27f561840aa1: Status 404 returned error can't find the container with id 7a0fc1891b3d07bc445bc5ed4ad9d97e83f78045f398dda6794a27f561840aa1
Oct 30 17:16:22.379078 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.379036    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j" event={"ID":"f7d643d6-6956-4378-aab5-b50f5f3cfc4a","Type":"ContainerStarted","Data":"7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af"}
Oct 30 17:16:22.379266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.379103    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j" event={"ID":"f7d643d6-6956-4378-aab5-b50f5f3cfc4a","Type":"ContainerStarted","Data":"6faf111ee8e28a1a98179684f67f90906ec08abaf92d409f0a71642c21469d82"}
Oct 30 17:16:22.379266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.379134    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:22.381060 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.381028    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47" event={"ID":"066e972e-b5f4-4778-985e-e4a3174f4bfb","Type":"ContainerStarted","Data":"fd7e81a5c229298c1d71da36fbc247cab8437a9e19e87202db296c678a871b46"}
Oct 30 17:16:22.381239 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.381087    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:22.381239 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.381108    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47" event={"ID":"066e972e-b5f4-4778-985e-e4a3174f4bfb","Type":"ContainerStarted","Data":"7a0fc1891b3d07bc445bc5ed4ad9d97e83f78045f398dda6794a27f561840aa1"}
Oct 30 17:16:22.415472 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.415371    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j" podStartSLOduration=2.415356548 podStartE2EDuration="2.415356548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:22.414247756 +0000 UTC m=+3212.268126020" watchObservedRunningTime="2026-10-30 17:16:22.415356548 +0000 UTC m=+3212.269234811"
Oct 30 17:16:22.475202 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.475119    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47" podStartSLOduration=2.475100977 podStartE2EDuration="2.475100977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:22.47169231 +0000 UTC m=+3212.325570682" watchObservedRunningTime="2026-10-30 17:16:22.475100977 +0000 UTC m=+3212.328979241"
Oct 30 17:16:22.498142 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.498112    4497 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 17:16:22.498290 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:22.498157    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 17:16:23.502929 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:23.502896    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:23.503431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:23.503414    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:23.504162 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:23.504139    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:16:23.810532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:23.810503    4497 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:16:23.810736 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:23.810559    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:16:24.716512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.716455    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 17:16:24.782264 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782228    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b81f7d-4567-46d4-a0b7-1647e183f814-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b81f7d-4567-46d4-a0b7-1647e183f814-istiod-ca-cert\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782444 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782297    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-credential-socket\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782444 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782357    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-certs\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782444 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782436    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-envoy\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782622 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782467    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-kube-api-access-bbjtr\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-kube-api-access-bbjtr\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782622 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782608    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-credential-socket" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:24.782731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782663    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b81f7d-4567-46d4-a0b7-1647e183f814-istiod-ca-cert" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:16:24.782731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782700    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-socket\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782838 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782750    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-podinfo\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782838 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782824    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-data\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782942 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782871    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-token\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-token\") pod \"d0b81f7d-4567-46d4-a0b7-1647e183f814\" (UID: \"d0b81f7d-4567-46d4-a0b7-1647e183f814\") "
Oct 30 17:16:24.782996 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782967    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-socket" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:24.783051 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.782988    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-certs" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:24.783325 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.783217    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-data" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:24.783767 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.783702    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:24.783767 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.783725    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:24.783767 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.783756    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0b81f7d-4567-46d4-a0b7-1647e183f814-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:24.783993 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.783772    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:24.783993 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.783785    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:24.785344 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.785321    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-kube-api-access-bbjtr" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "kube-api-access-bbjtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:24.786075 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.786004    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-envoy" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:16:24.786789 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.786765    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-podinfo" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:16:24.786789 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.786777    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-token" pod "d0b81f7d-4567-46d4-a0b7-1647e183f814" (UID: "d0b81f7d-4567-46d4-a0b7-1647e183f814"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:16:24.885647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.885615    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:24.885647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.885645    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:24.885863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.885654    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbjtr\" (UniqueName: \"kubernetes.io/projected/d0b81f7d-4567-46d4-a0b7-1647e183f814-kube-api-access-bbjtr\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:24.885863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:24.885663    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0b81f7d-4567-46d4-a0b7-1647e183f814-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:16:25.404543 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.404508    4497 generic.go:356] "Generic (PLEG): container finished" podID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerID="d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292" exitCode=0
Oct 30 17:16:25.404729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.404561    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"
Oct 30 17:16:25.404729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.404590    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" event={"ID":"d0b81f7d-4567-46d4-a0b7-1647e183f814","Type":"ContainerDied","Data":"d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292"}
Oct 30 17:16:25.404729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.404647    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6" event={"ID":"d0b81f7d-4567-46d4-a0b7-1647e183f814","Type":"ContainerDied","Data":"4edf8acd1e6da5dcc99bd31e36a75a8ba42259bec4e47b8b826e50fcc1b40211"}
Oct 30 17:16:25.404729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.404674    4497 scope.go:122] "RemoveContainer" containerID="d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292"
Oct 30 17:16:25.417523 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.417507    4497 scope.go:122] "RemoveContainer" containerID="d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292"
Oct 30 17:16:25.417805 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:16:25.417774    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292\": container with ID starting with d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292 not found: ID does not exist" containerID="d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292"
Oct 30 17:16:25.417860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.417814    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292"} err="failed to get container status \"d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292\": rpc error: code = NotFound desc = could not find container \"d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292\": container with ID starting with d9e0e2374706b2d1cf901514de477eccb1846332ddf6235bff79c82910452292 not found: ID does not exist"
Oct 30 17:16:25.463808 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.463774    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"]
Oct 30 17:16:25.472852 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:25.472821    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-j6rj6"]
Oct 30 17:16:26.305383 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:26.302828    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" path="/var/lib/kubelet/pods/d0b81f7d-4567-46d4-a0b7-1647e183f814/volumes"
Oct 30 17:16:28.343688 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:28.343658    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:16:33.391198 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:33.391146    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:16:33.391675 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:33.391241    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:16:50.826073 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.826031    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"]
Oct 30 17:16:50.828407 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.828377    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy"
Oct 30 17:16:50.828580 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.828565    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy"
Oct 30 17:16:50.829109 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.829090    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b81f7d-4567-46d4-a0b7-1647e183f814" containerName="istio-proxy"
Oct 30 17:16:50.836296 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.836275    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:50.848419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.846500    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"]
Oct 30 17:16:50.850430 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.850406    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bbc36f-serving-cert\""
Oct 30 17:16:50.984471 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.984436    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z624l\" (UniqueName: \"kubernetes.io/projected/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-kube-api-access-z624l\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:50.984730 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.984565    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-maas-api-tls\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:50.984806 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.984750    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-metrics-certs\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:50.984806 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:50.984798    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.086633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.086547    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z624l\" (UniqueName: \"kubernetes.io/projected/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-kube-api-access-z624l\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.086633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.086629    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-maas-api-tls\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.086974 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.086754    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-metrics-certs\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.086974 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.086801    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.087524 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.087495    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.089259 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.089227    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-maas-api-tls\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.089365 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.089231    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-metrics-certs\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.109027 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.108985    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z624l\" (UniqueName: \"kubernetes.io/projected/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-kube-api-access-z624l\") pod \"maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.154179 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.154145    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:51.468373 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.468351    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"]
Oct 30 17:16:51.470831 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:16:51.470803    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec157bad_0aa9_4abe_96fc_f1e7b7a4186e.slice/crio-075238c75612f048ce9bcfbf49619bfbed1b1c6a4da46c8e728bea2b441db464 WatchSource:0}: Error finding container 075238c75612f048ce9bcfbf49619bfbed1b1c6a4da46c8e728bea2b441db464: Status 404 returned error can't find the container with id 075238c75612f048ce9bcfbf49619bfbed1b1c6a4da46c8e728bea2b441db464
Oct 30 17:16:51.611245 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:51.611212    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7" event={"ID":"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e","Type":"ContainerStarted","Data":"075238c75612f048ce9bcfbf49619bfbed1b1c6a4da46c8e728bea2b441db464"}
Oct 30 17:16:51.916519 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:16:51.916482    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:16:52.622221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:52.622157    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7" event={"ID":"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e","Type":"ContainerStarted","Data":"cd5fc009c510bba96c716b270b746f2b51b4bdeb04a65a0145b34d9a98d51c56"}
Oct 30 17:16:52.661255 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:52.661204    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7" podStartSLOduration=2.350880938 podStartE2EDuration="2.661178602s" totalImagesPullingTime="310.297664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:52.659145344 +0000 UTC m=+3242.513023590" watchObservedRunningTime="2026-10-30 17:16:52.661178602 +0000 UTC m=+3242.515056865"
Oct 30 17:16:53.630848 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:53.630812    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:16:54.629370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.629338    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"]
Oct 30 17:16:54.634057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.634036    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.651670 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.651645    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4c6596-serving-cert\""
Oct 30 17:16:54.667170 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.667137    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"]
Oct 30 17:16:54.750023 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.749981    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-metrics-certs\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.750422 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.750397    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2gx\" (UniqueName: \"kubernetes.io/projected/6ecefbcb-2750-4aba-ab53-12d644a896f5-kube-api-access-2v2gx\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.750583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.750565    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecefbcb-2750-4aba-ab53-12d644a896f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.750741 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.750727    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.852976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.852938    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2gx\" (UniqueName: \"kubernetes.io/projected/6ecefbcb-2750-4aba-ab53-12d644a896f5-kube-api-access-2v2gx\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.852976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.852981    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecefbcb-2750-4aba-ab53-12d644a896f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.853267 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.853021    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.853337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.853306    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-metrics-certs\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.853698 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.853673    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecefbcb-2750-4aba-ab53-12d644a896f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.855715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.855690    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-metrics-certs\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.855881 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.855859    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.875164 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.875133    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2gx\" (UniqueName: \"kubernetes.io/projected/6ecefbcb-2750-4aba-ab53-12d644a896f5-kube-api-access-2v2gx\") pod \"maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:54.955276 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:54.953848    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:55.392304 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:55.392269    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"]
Oct 30 17:16:55.394908 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:16:55.394878    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecefbcb_2750_4aba_ab53_12d644a896f5.slice/crio-a827ec6e94ca9a8e8ba412356c08be60f236c5469d24ec5ecb4d09ca3f63c665 WatchSource:0}: Error finding container a827ec6e94ca9a8e8ba412356c08be60f236c5469d24ec5ecb4d09ca3f63c665: Status 404 returned error can't find the container with id a827ec6e94ca9a8e8ba412356c08be60f236c5469d24ec5ecb4d09ca3f63c665
Oct 30 17:16:55.396523 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:55.396504    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:16:55.652062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:55.651984    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf" event={"ID":"6ecefbcb-2750-4aba-ab53-12d644a896f5","Type":"ContainerStarted","Data":"a827ec6e94ca9a8e8ba412356c08be60f236c5469d24ec5ecb4d09ca3f63c665"}
Oct 30 17:16:56.663036 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:56.662998    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf" event={"ID":"6ecefbcb-2750-4aba-ab53-12d644a896f5","Type":"ContainerStarted","Data":"27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7"}
Oct 30 17:16:56.663452 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:56.663092    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:16:56.706614 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:56.706546    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf" podStartSLOduration=2.062971967 podStartE2EDuration="2.706529673s" totalImagesPullingTime="643.557706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:56.702576331 +0000 UTC m=+3246.556454596" watchObservedRunningTime="2026-10-30 17:16:56.706529673 +0000 UTC m=+3246.560407941"
Oct 30 17:16:59.644640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:16:59.644611    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:17:02.676863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:17:02.676835    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:17:21.918048 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:17:21.918013    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:17:51.922137 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:17:51.918617    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:18:21.917790 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:18:21.917751    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:18:24.084824 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.084790    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"]
Oct 30 17:18:24.089809 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.089788    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:24.102970 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.102951    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0b09d3\""
Oct 30 17:18:24.111098 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.111073    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"]
Oct 30 17:18:24.113010 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.112988    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0b09d3-dockercfg-f8px6\""
Oct 30 17:18:24.134783 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.134747    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2592437d-ef7a-48d5-a319-35fe7287197d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl\" (UID: \"2592437d-ef7a-48d5-a319-35fe7287197d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:24.134999 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.134862    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bhh\" (UniqueName: \"kubernetes.io/projected/2592437d-ef7a-48d5-a319-35fe7287197d-kube-api-access-v9bhh\") pod \"payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl\" (UID: \"2592437d-ef7a-48d5-a319-35fe7287197d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:24.270539 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.270486    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2592437d-ef7a-48d5-a319-35fe7287197d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl\" (UID: \"2592437d-ef7a-48d5-a319-35fe7287197d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:24.270944 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.270919    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bhh\" (UniqueName: \"kubernetes.io/projected/2592437d-ef7a-48d5-a319-35fe7287197d-kube-api-access-v9bhh\") pod \"payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl\" (UID: \"2592437d-ef7a-48d5-a319-35fe7287197d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:24.272958 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.272932    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2592437d-ef7a-48d5-a319-35fe7287197d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl\" (UID: \"2592437d-ef7a-48d5-a319-35fe7287197d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:24.353299 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.352959    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bhh\" (UniqueName: \"kubernetes.io/projected/2592437d-ef7a-48d5-a319-35fe7287197d-kube-api-access-v9bhh\") pod \"payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl\" (UID: \"2592437d-ef7a-48d5-a319-35fe7287197d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:24.393624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.393554    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"]
Oct 30 17:18:24.400349 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.400319    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"]
Oct 30 17:18:24.400827 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.400719    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:24.406800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.404491    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:24.474513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.474465    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e622cd5-6f48-4de4-af56-afd3a47279c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9\" (UID: \"0e622cd5-6f48-4de4-af56-afd3a47279c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:24.474691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.474672    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89w2l\" (UniqueName: \"kubernetes.io/projected/0e622cd5-6f48-4de4-af56-afd3a47279c5-kube-api-access-89w2l\") pod \"payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9\" (UID: \"0e622cd5-6f48-4de4-af56-afd3a47279c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:24.576867 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.576599    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w2l\" (UniqueName: \"kubernetes.io/projected/0e622cd5-6f48-4de4-af56-afd3a47279c5-kube-api-access-89w2l\") pod \"payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9\" (UID: \"0e622cd5-6f48-4de4-af56-afd3a47279c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:24.576867 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.576780    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e622cd5-6f48-4de4-af56-afd3a47279c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9\" (UID: \"0e622cd5-6f48-4de4-af56-afd3a47279c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:24.578221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.578155    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e622cd5-6f48-4de4-af56-afd3a47279c5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9\" (UID: \"0e622cd5-6f48-4de4-af56-afd3a47279c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:24.599230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.599157    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89w2l\" (UniqueName: \"kubernetes.io/projected/0e622cd5-6f48-4de4-af56-afd3a47279c5-kube-api-access-89w2l\") pod \"payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9\" (UID: \"0e622cd5-6f48-4de4-af56-afd3a47279c5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:24.744642 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.744544    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:24.751958 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:24.751930    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"]
Oct 30 17:18:24.760056 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:18:24.760023    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2592437d_ef7a_48d5_a319_35fe7287197d.slice/crio-f192c31d58d95024a507cbcd8ab30e8c4b8c550b110457cdc22cb2247ef7dd82 WatchSource:0}: Error finding container f192c31d58d95024a507cbcd8ab30e8c4b8c550b110457cdc22cb2247ef7dd82: Status 404 returned error can't find the container with id f192c31d58d95024a507cbcd8ab30e8c4b8c550b110457cdc22cb2247ef7dd82
Oct 30 17:18:25.098932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.098901    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"]
Oct 30 17:18:25.101595 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:18:25.101563    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e622cd5_6f48_4de4_af56_afd3a47279c5.slice/crio-baf1f39c9ddd93eabb5050a72d0c30d6ad40f4b37837fdfee267f7cacf7b2824 WatchSource:0}: Error finding container baf1f39c9ddd93eabb5050a72d0c30d6ad40f4b37837fdfee267f7cacf7b2824: Status 404 returned error can't find the container with id baf1f39c9ddd93eabb5050a72d0c30d6ad40f4b37837fdfee267f7cacf7b2824
Oct 30 17:18:25.531180 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.531145    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9" event={"ID":"0e622cd5-6f48-4de4-af56-afd3a47279c5","Type":"ContainerStarted","Data":"99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447"}
Oct 30 17:18:25.531401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.531238    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9" event={"ID":"0e622cd5-6f48-4de4-af56-afd3a47279c5","Type":"ContainerStarted","Data":"baf1f39c9ddd93eabb5050a72d0c30d6ad40f4b37837fdfee267f7cacf7b2824"}
Oct 30 17:18:25.531401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.531273    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:25.532960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.532937    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl" event={"ID":"2592437d-ef7a-48d5-a319-35fe7287197d","Type":"ContainerStarted","Data":"82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172"}
Oct 30 17:18:25.533080 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.532974    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl" event={"ID":"2592437d-ef7a-48d5-a319-35fe7287197d","Type":"ContainerStarted","Data":"f192c31d58d95024a507cbcd8ab30e8c4b8c550b110457cdc22cb2247ef7dd82"}
Oct 30 17:18:25.533080 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.533025    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:25.568302 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.568262    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9" podStartSLOduration=1.568249577 podStartE2EDuration="1.568249577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:25.565869862 +0000 UTC m=+3335.419748127" watchObservedRunningTime="2026-10-30 17:18:25.568249577 +0000 UTC m=+3335.422127842"
Oct 30 17:18:25.602042 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:25.602001    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl" podStartSLOduration=1.6019885980000002 podStartE2EDuration="1.601988598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:25.600701806 +0000 UTC m=+3335.454580070" watchObservedRunningTime="2026-10-30 17:18:25.601988598 +0000 UTC m=+3335.455866863"
Oct 30 17:18:36.543793 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:36.543763    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:18:36.544307 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:18:36.543895    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:18:51.940138 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:18:51.940101    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:19:18.284082 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.284040    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"]
Oct 30 17:19:18.290808 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.290784    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.307749 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.307726    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e15d1f-openshift-default-dockercfg-jgp98\""
Oct 30 17:19:18.322865 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.322840    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"]
Oct 30 17:19:18.366908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.366874    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-istio-token\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.367086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.367003    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-data\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.367086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.367038    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0191197b-1734-4678-9304-7bfff05b46ef-istio-podinfo\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.367250 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.367126    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-certs\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.367417 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.367393    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0191197b-1734-4678-9304-7bfff05b46ef-istiod-ca-cert\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.367516 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.367499    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-envoy\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.367741 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.367723    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.367821 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.367782    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb88k\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-kube-api-access-rb88k\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.367887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.367864    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-credential-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.469472 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469448    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-data\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.469618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469478    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0191197b-1734-4678-9304-7bfff05b46ef-istio-podinfo\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.469618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469502    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-certs\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.469618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469595    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0191197b-1734-4678-9304-7bfff05b46ef-istiod-ca-cert\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.469788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469686    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-envoy\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.469788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469723    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.469908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469789    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb88k\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-kube-api-access-rb88k\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.469908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469891    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-credential-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.470023 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469958    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-istio-token\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.470023 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469982    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-certs\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.470151 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.469956    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-data\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.470312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.470291    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-credential-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.470399 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.470341    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0191197b-1734-4678-9304-7bfff05b46ef-istiod-ca-cert\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.470399 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.470355    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.472102 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.472079    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-envoy\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.472302 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.472284    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0191197b-1734-4678-9304-7bfff05b46ef-istio-podinfo\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.490770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.490742    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-istio-token\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.490908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.490887    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb88k\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-kube-api-access-rb88k\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.605541 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.605463    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:18.935795 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.935766    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"]
Oct 30 17:19:18.937837 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:19:18.937811    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0191197b_1734_4678_9304_7bfff05b46ef.slice/crio-2a9730b5df3e3b4d1ce024642d37b8b25b5e7770c717689c3e268566c46aab5b WatchSource:0}: Error finding container 2a9730b5df3e3b4d1ce024642d37b8b25b5e7770c717689c3e268566c46aab5b: Status 404 returned error can't find the container with id 2a9730b5df3e3b4d1ce024642d37b8b25b5e7770c717689c3e268566c46aab5b
Oct 30 17:19:18.940958 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.940927    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:19:18.941037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.940991    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:19:18.941037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:18.941031    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:19:19.046056 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:19.046027    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" event={"ID":"0191197b-1734-4678-9304-7bfff05b46ef","Type":"ContainerStarted","Data":"856ffacb7068817509bc3fad08aa22c91038ae5ae279f6aef8b9cd9d40bf42e3"}
Oct 30 17:19:19.046211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:19.046086    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" event={"ID":"0191197b-1734-4678-9304-7bfff05b46ef","Type":"ContainerStarted","Data":"2a9730b5df3e3b4d1ce024642d37b8b25b5e7770c717689c3e268566c46aab5b"}
Oct 30 17:19:19.094787 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:19.094706    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" podStartSLOduration=1.094685019 podStartE2EDuration="1.094685019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:19.090214349 +0000 UTC m=+3388.944092637" watchObservedRunningTime="2026-10-30 17:19:19.094685019 +0000 UTC m=+3388.948563292"
Oct 30 17:19:19.610366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:19.610330    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:20.608264 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:20.608181    4497 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 17:19:20.608476 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:20.608309    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 17:19:20.930917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:20.930844    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"]
Oct 30 17:19:20.931362 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:20.931030    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy" containerID="cri-o://81b7dc16b40a246b9037a88936e3709bf76726bf5e61d73f282daa1d6e16fa48" gracePeriod=30
Oct 30 17:19:21.606146 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:21.606105    4497 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 17:19:21.606389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:21.606160    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 17:19:21.924108 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:19:21.924022    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:19:22.605989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:22.605951    4497 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 17:19:22.606424 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:22.606010    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 17:19:23.606089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:23.606053    4497 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 17:19:23.606584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:23.606111    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 17:19:24.606655 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:24.606623    4497 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Oct 30 17:19:24.607079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:24.606673    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Oct 30 17:19:25.612063 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:25.612033    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:25.612506 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:25.612486    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:25.613166 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:25.613144    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:19:26.140523 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.140485    4497 generic.go:356] "Generic (PLEG): container finished" podID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerID="81b7dc16b40a246b9037a88936e3709bf76726bf5e61d73f282daa1d6e16fa48" exitCode=0
Oct 30 17:19:26.141085 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.141050    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" event={"ID":"f019fe5f-39a5-4658-996c-4f91a78231e8","Type":"ContainerDied","Data":"81b7dc16b40a246b9037a88936e3709bf76726bf5e61d73f282daa1d6e16fa48"}
Oct 30 17:19:26.141329 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.141132    4497 scope.go:122] "RemoveContainer" containerID="846072f3c384abf7d12ce69f59a2c2022e06b9953760f5463b561da350194d11"
Oct 30 17:19:26.222384 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.222323    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 17:19:26.343388 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343357    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-token\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-token\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.343576 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343414    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-kube-api-access-8xd9p\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-kube-api-access-8xd9p\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.343576 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343509    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-socket\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.343696 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343599    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-podinfo\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.343696 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343650    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f019fe5f-39a5-4658-996c-4f91a78231e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019fe5f-39a5-4658-996c-4f91a78231e8-istiod-ca-cert\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.343806 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343766    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-envoy\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.343862 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343808    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-certs\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.343862 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343824    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-socket" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:26.343966 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.343952    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-credential-socket\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.344064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.344032    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-data\") pod \"f019fe5f-39a5-4658-996c-4f91a78231e8\" (UID: \"f019fe5f-39a5-4658-996c-4f91a78231e8\") "
Oct 30 17:19:26.344379 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.344350    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-certs" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:26.344379 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.344365    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-credential-socket" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:26.344654 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.344635    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-data" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:26.344729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.344554    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f019fe5f-39a5-4658-996c-4f91a78231e8-istiod-ca-cert" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:19:26.345539 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.345509    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.345640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.345542    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.345640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.345556    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019fe5f-39a5-4658-996c-4f91a78231e8-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.345640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.345572    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.345640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.345586    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.346598 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.346571    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-kube-api-access-8xd9p" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "kube-api-access-8xd9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:19:26.346744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.346609    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-token" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:19:26.346817 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.346789    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-podinfo" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:19:26.346943 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.346918    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-envoy" pod "f019fe5f-39a5-4658-996c-4f91a78231e8" (UID: "f019fe5f-39a5-4658-996c-4f91a78231e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:26.447294 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.447254    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.447294 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.447289    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xd9p\" (UniqueName: \"kubernetes.io/projected/f019fe5f-39a5-4658-996c-4f91a78231e8-kube-api-access-8xd9p\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.447294 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.447301    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.447598 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.447310    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019fe5f-39a5-4658-996c-4f91a78231e8-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:26.502921 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.502853    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"]
Oct 30 17:19:26.535580 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:26.535543    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"]
Oct 30 17:19:27.271213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:27.266589    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"
Oct 30 17:19:27.271213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:27.267273    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj" event={"ID":"f019fe5f-39a5-4658-996c-4f91a78231e8","Type":"ContainerDied","Data":"44de95dcaf98361d06a4d894ca863124dbcde80e0ae25dba314cfd5608c512ac"}
Oct 30 17:19:27.271213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:27.267353    4497 scope.go:122] "RemoveContainer" containerID="81b7dc16b40a246b9037a88936e3709bf76726bf5e61d73f282daa1d6e16fa48"
Oct 30 17:19:27.392880 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:27.390394    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"]
Oct 30 17:19:27.399123 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:27.399091    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-lpwcj"]
Oct 30 17:19:28.303347 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:28.303316    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" path="/var/lib/kubelet/pods/f019fe5f-39a5-4658-996c-4f91a78231e8/volumes"
Oct 30 17:19:31.503474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:31.503437    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4" podUID="be939ad3-346e-4401-a5d8-65d4f94d46aa" containerName="payload-processing" containerID="cri-o://60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e" gracePeriod=25
Oct 30 17:19:31.536022 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:31.535992    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw" podUID="5a95d1b9-44a3-4292-a1bf-b2e974fffe81" containerName="payload-pre-processing" containerID="cri-o://8ce1988e9d750fd13db3eee12e0bf854ebb11b808baa2b36101254a58632f97d" gracePeriod=25
Oct 30 17:19:31.792355 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:31.792331    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:19:31.966159 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:31.966124    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be939ad3-346e-4401-a5d8-65d4f94d46aa-kube-api-access-5fx6d\" (UniqueName: \"kubernetes.io/projected/be939ad3-346e-4401-a5d8-65d4f94d46aa-kube-api-access-5fx6d\") pod \"be939ad3-346e-4401-a5d8-65d4f94d46aa\" (UID: \"be939ad3-346e-4401-a5d8-65d4f94d46aa\") "
Oct 30 17:19:31.966390 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:31.966370    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be939ad3-346e-4401-a5d8-65d4f94d46aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be939ad3-346e-4401-a5d8-65d4f94d46aa-plugins-config-volume\") pod \"be939ad3-346e-4401-a5d8-65d4f94d46aa\" (UID: \"be939ad3-346e-4401-a5d8-65d4f94d46aa\") "
Oct 30 17:19:31.966744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:31.966710    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be939ad3-346e-4401-a5d8-65d4f94d46aa-plugins-config-volume" pod "be939ad3-346e-4401-a5d8-65d4f94d46aa" (UID: "be939ad3-346e-4401-a5d8-65d4f94d46aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:19:31.967065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:31.967049    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be939ad3-346e-4401-a5d8-65d4f94d46aa-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:31.968280 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:31.968255    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be939ad3-346e-4401-a5d8-65d4f94d46aa-kube-api-access-5fx6d" pod "be939ad3-346e-4401-a5d8-65d4f94d46aa" (UID: "be939ad3-346e-4401-a5d8-65d4f94d46aa"). InnerVolumeSpecName "kube-api-access-5fx6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:19:32.068661 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.068593    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fx6d\" (UniqueName: \"kubernetes.io/projected/be939ad3-346e-4401-a5d8-65d4f94d46aa-kube-api-access-5fx6d\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:32.164445 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.164412    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw" podUID="5a95d1b9-44a3-4292-a1bf-b2e974fffe81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:19:32.307783 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.307747    4497 generic.go:356] "Generic (PLEG): container finished" podID="be939ad3-346e-4401-a5d8-65d4f94d46aa" containerID="60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e" exitCode=0
Oct 30 17:19:32.307783 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.307786    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"
Oct 30 17:19:32.308021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.307821    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4" event={"ID":"be939ad3-346e-4401-a5d8-65d4f94d46aa","Type":"ContainerDied","Data":"60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e"}
Oct 30 17:19:32.308021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.307872    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4" event={"ID":"be939ad3-346e-4401-a5d8-65d4f94d46aa","Type":"ContainerDied","Data":"c51f87a153588a155cdea2e982c5a369b6756466a3b09195383cd54e21d19c72"}
Oct 30 17:19:32.308021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.307898    4497 scope.go:122] "RemoveContainer" containerID="60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e"
Oct 30 17:19:32.321752 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.321696    4497 scope.go:122] "RemoveContainer" containerID="60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e"
Oct 30 17:19:32.321975 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:19:32.321943    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e\": container with ID starting with 60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e not found: ID does not exist" containerID="60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e"
Oct 30 17:19:32.322111 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.321981    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e"} err="failed to get container status \"60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e\": rpc error: code = NotFound desc = could not find container \"60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e\": container with ID starting with 60fd1cc352566950cbe97bc57a632708fc17aaf10f623e0b94d9e2844eb2585e not found: ID does not exist"
Oct 30 17:19:32.354807 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.354778    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"]
Oct 30 17:19:32.381835 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:32.381807    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6acc78-58d69d49d4-5fdh4"]
Oct 30 17:19:34.302205 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:34.302151    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be939ad3-346e-4401-a5d8-65d4f94d46aa" path="/var/lib/kubelet/pods/be939ad3-346e-4401-a5d8-65d4f94d46aa/volumes"
Oct 30 17:19:34.906977 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:34.906944    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"]
Oct 30 17:19:34.907160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:34.907135    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy" containerID="cri-o://9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400" gracePeriod=30
Oct 30 17:19:35.643261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:35.643232    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"]
Oct 30 17:19:35.643678 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:35.643439    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy" containerID="cri-o://01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810" gracePeriod=30
Oct 30 17:19:40.204938 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.204913    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:19:40.299422 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299377    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-socket\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.299616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299452    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-token\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-token\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.299616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299595    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-kube-api-access-zrxcx\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-kube-api-access-zrxcx\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.299616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299617    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-credential-socket\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.299811 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299643    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-podinfo\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.299811 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299736    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-envoy\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.299811 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299771    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c06ac1a-0119-4dde-a02a-8ab491565a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06ac1a-0119-4dde-a02a-8ab491565a05-istiod-ca-cert\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.299993 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299826    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-certs\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.299993 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.299869    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-data\") pod \"8c06ac1a-0119-4dde-a02a-8ab491565a05\" (UID: \"8c06ac1a-0119-4dde-a02a-8ab491565a05\") "
Oct 30 17:19:40.300423 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.300395    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-credential-socket" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:40.300769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.300744    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-data" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:40.300953 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.300933    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-certs" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:40.301147 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.301126    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-socket" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:40.301274 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.301243    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c06ac1a-0119-4dde-a02a-8ab491565a05-istiod-ca-cert" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:19:40.303577 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.303556    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-envoy" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:40.308624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.308591    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-kube-api-access-zrxcx" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "kube-api-access-zrxcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:19:40.308727 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.308684    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-token" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:19:40.308867 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.308843    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-podinfo" pod "8c06ac1a-0119-4dde-a02a-8ab491565a05" (UID: "8c06ac1a-0119-4dde-a02a-8ab491565a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:19:40.379658 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.379611    4497 generic.go:356] "Generic (PLEG): container finished" podID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerID="9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400" exitCode=0
Oct 30 17:19:40.379860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.379811    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" event={"ID":"8c06ac1a-0119-4dde-a02a-8ab491565a05","Type":"ContainerDied","Data":"9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400"}
Oct 30 17:19:40.379860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.379855    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n" event={"ID":"8c06ac1a-0119-4dde-a02a-8ab491565a05","Type":"ContainerDied","Data":"491ff791c6b2aa04584bd79c4426712df219ff4e14ebbc262b5eeab908a389d8"}
Oct 30 17:19:40.379977 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.379880    4497 scope.go:122] "RemoveContainer" containerID="9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400"
Oct 30 17:19:40.380022 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.380013    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"
Oct 30 17:19:40.399801 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.399775    4497 scope.go:122] "RemoveContainer" containerID="9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400"
Oct 30 17:19:40.400089 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:19:40.400071    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400\": container with ID starting with 9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400 not found: ID does not exist" containerID="9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400"
Oct 30 17:19:40.400137 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.400098    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400"} err="failed to get container status \"9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400\": rpc error: code = NotFound desc = could not find container \"9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400\": container with ID starting with 9fb6316207943926969103a60edce772cf941c8bb2dcd86663826ccd441ee400 not found: ID does not exist"
Oct 30 17:19:40.402361 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402331    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.402361 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402355    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxcx\" (UniqueName: \"kubernetes.io/projected/8c06ac1a-0119-4dde-a02a-8ab491565a05-kube-api-access-zrxcx\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.402361 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402365    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.402602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402374    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.402602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402392    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.402602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402401    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c06ac1a-0119-4dde-a02a-8ab491565a05-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.402602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402411    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.402602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402422    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.402602 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.402434    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c06ac1a-0119-4dde-a02a-8ab491565a05-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:40.439141 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.439083    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"]
Oct 30 17:19:40.444463 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.444440    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x8n9n"]
Oct 30 17:19:40.899113 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:40.899084    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:19:41.010249 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010212    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-socket\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.010451 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010262    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-credential-socket\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.010451 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010375    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-kube-api-access-xd4t6\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-kube-api-access-xd4t6\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.010451 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010392    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-envoy\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.010451 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010429    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-certs\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.010689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010493    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-socket" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:41.010689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010512    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-data\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.010689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010586    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-podinfo\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.010689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010611    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-credential-socket" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:41.010689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010620    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-token\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.011064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010747    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-certs" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:41.011064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010752    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istiod-ca-cert\") pod \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\" (UID: \"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9\") "
Oct 30 17:19:41.011064 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.010783    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-data" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:41.011240 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.011061    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istiod-ca-cert" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:19:41.011562 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.011543    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.011693 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.011565    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.011693 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.011577    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.011693 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.011589    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.011693 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.011603    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.013261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.013172    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-kube-api-access-xd4t6" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "kube-api-access-xd4t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:19:41.013261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.013249    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-token" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:19:41.013421 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.013265    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-podinfo" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:19:41.013421 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.013277    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-envoy" pod "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" (UID: "ca0e40c4-79ba-44c1-b762-ab8dc252fbd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:19:41.118459 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.118422    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd4t6\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-kube-api-access-xd4t6\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.118699 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.118683    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.118804 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.118793    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.118891 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.118881    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:41.389314 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.389235    4497 generic.go:356] "Generic (PLEG): container finished" podID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerID="01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810" exitCode=0
Oct 30 17:19:41.389314 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.389292    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"
Oct 30 17:19:41.389731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.389320    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" event={"ID":"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9","Type":"ContainerDied","Data":"01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810"}
Oct 30 17:19:41.389731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.389370    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n" event={"ID":"ca0e40c4-79ba-44c1-b762-ab8dc252fbd9","Type":"ContainerDied","Data":"0932fe52e86df4fa36691716620ada6470c9366326a65e08dfb246429f61eb1e"}
Oct 30 17:19:41.389731 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.389392    4497 scope.go:122] "RemoveContainer" containerID="01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810"
Oct 30 17:19:41.401007 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.400991    4497 scope.go:122] "RemoveContainer" containerID="01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810"
Oct 30 17:19:41.401286 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:19:41.401262    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810\": container with ID starting with 01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810 not found: ID does not exist" containerID="01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810"
Oct 30 17:19:41.401419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.401289    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810"} err="failed to get container status \"01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810\": rpc error: code = NotFound desc = could not find container \"01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810\": container with ID starting with 01ab439a7ce587c1d4d4876c822f7ba8043a351896d3471cdbdb6963946f5810 not found: ID does not exist"
Oct 30 17:19:41.436132 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.436108    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"]
Oct 30 17:19:41.448080 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:41.448061    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4h68n"]
Oct 30 17:19:42.164142 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:42.164099    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw" podUID="5a95d1b9-44a3-4292-a1bf-b2e974fffe81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:19:42.422260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:42.411545    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" path="/var/lib/kubelet/pods/8c06ac1a-0119-4dde-a02a-8ab491565a05/volumes"
Oct 30 17:19:42.422260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:42.412468    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" path="/var/lib/kubelet/pods/ca0e40c4-79ba-44c1-b762-ab8dc252fbd9/volumes"
Oct 30 17:19:51.823682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:51.823656    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:19:51.850758 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:51.850732    4497 scope.go:122] "RemoveContainer" containerID="8ce1988e9d750fd13db3eee12e0bf854ebb11b808baa2b36101254a58632f97d"
Oct 30 17:19:51.924546 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:19:51.924507    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:19:52.000232 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.000180    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-plugins-config-volume\") pod \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\" (UID: \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\") "
Oct 30 17:19:52.000458 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.000446    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-kube-api-access-pfldz\" (UniqueName: \"kubernetes.io/projected/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-kube-api-access-pfldz\") pod \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\" (UID: \"5a95d1b9-44a3-4292-a1bf-b2e974fffe81\") "
Oct 30 17:19:52.000571 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.000546    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-plugins-config-volume" pod "5a95d1b9-44a3-4292-a1bf-b2e974fffe81" (UID: "5a95d1b9-44a3-4292-a1bf-b2e974fffe81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:19:52.000909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.000897    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:52.002445 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.002424    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-kube-api-access-pfldz" pod "5a95d1b9-44a3-4292-a1bf-b2e974fffe81" (UID: "5a95d1b9-44a3-4292-a1bf-b2e974fffe81"). InnerVolumeSpecName "kube-api-access-pfldz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:19:52.102343 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.102278    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfldz\" (UniqueName: \"kubernetes.io/projected/5a95d1b9-44a3-4292-a1bf-b2e974fffe81-kube-api-access-pfldz\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:19:52.541568 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.541533    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw" event={"ID":"5a95d1b9-44a3-4292-a1bf-b2e974fffe81","Type":"ContainerDied","Data":"8ce1988e9d750fd13db3eee12e0bf854ebb11b808baa2b36101254a58632f97d"}
Oct 30 17:19:52.541734 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.541595    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw" event={"ID":"5a95d1b9-44a3-4292-a1bf-b2e974fffe81","Type":"ContainerDied","Data":"79301d8792fe370e1af5f2fa8d986baffde3d0fd85ef64e4984ee5d1d6a76002"}
Oct 30 17:19:52.541734 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.541543    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"
Oct 30 17:19:52.612727 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.612689    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"]
Oct 30 17:19:52.617998 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:52.617971    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6acc78-6675cfb655-s9rfw"]
Oct 30 17:19:54.303824 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:19:54.303793    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a95d1b9-44a3-4292-a1bf-b2e974fffe81" path="/var/lib/kubelet/pods/5a95d1b9-44a3-4292-a1bf-b2e974fffe81/volumes"
Oct 30 17:20:06.546838 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.546802    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"]
Oct 30 17:20:06.547361 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.547069    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6" podUID="153210bf-399a-41bd-a5e3-de2a38b7d549" containerName="maas-api" containerID="cri-o://dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241" gracePeriod=30
Oct 30 17:20:06.570377 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.570338    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6" podUID="153210bf-399a-41bd-a5e3-de2a38b7d549" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.218:8443/health\": dial tcp 10.133.0.218:8443: connect: connection refused"
Oct 30 17:20:06.886911 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.886879    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:20:06.944764 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.944729    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153210bf-399a-41bd-a5e3-de2a38b7d549-kube-api-access-5vfrj\" (UniqueName: \"kubernetes.io/projected/153210bf-399a-41bd-a5e3-de2a38b7d549-kube-api-access-5vfrj\") pod \"153210bf-399a-41bd-a5e3-de2a38b7d549\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") "
Oct 30 17:20:06.944963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.944791    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-metrics-certs\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-metrics-certs\") pod \"153210bf-399a-41bd-a5e3-de2a38b7d549\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") "
Oct 30 17:20:06.944963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.944917    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153210bf-399a-41bd-a5e3-de2a38b7d549-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153210bf-399a-41bd-a5e3-de2a38b7d549-openshift-service-ca\") pod \"153210bf-399a-41bd-a5e3-de2a38b7d549\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") "
Oct 30 17:20:06.945108 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.944998    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-maas-api-tls\") pod \"153210bf-399a-41bd-a5e3-de2a38b7d549\" (UID: \"153210bf-399a-41bd-a5e3-de2a38b7d549\") "
Oct 30 17:20:06.945403 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.945372    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153210bf-399a-41bd-a5e3-de2a38b7d549-openshift-service-ca" pod "153210bf-399a-41bd-a5e3-de2a38b7d549" (UID: "153210bf-399a-41bd-a5e3-de2a38b7d549"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:20:06.945803 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.945749    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153210bf-399a-41bd-a5e3-de2a38b7d549-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:20:06.947908 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.947885    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-metrics-certs" pod "153210bf-399a-41bd-a5e3-de2a38b7d549" (UID: "153210bf-399a-41bd-a5e3-de2a38b7d549"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:20:06.972403 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.972315    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-maas-api-tls" pod "153210bf-399a-41bd-a5e3-de2a38b7d549" (UID: "153210bf-399a-41bd-a5e3-de2a38b7d549"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:20:06.972591 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:06.972486    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153210bf-399a-41bd-a5e3-de2a38b7d549-kube-api-access-5vfrj" pod "153210bf-399a-41bd-a5e3-de2a38b7d549" (UID: "153210bf-399a-41bd-a5e3-de2a38b7d549"). InnerVolumeSpecName "kube-api-access-5vfrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:07.047023 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.046990    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vfrj\" (UniqueName: \"kubernetes.io/projected/153210bf-399a-41bd-a5e3-de2a38b7d549-kube-api-access-5vfrj\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:20:07.047023 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.047018    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:20:07.047023 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.047029    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153210bf-399a-41bd-a5e3-de2a38b7d549-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:20:07.672267 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.672235    4497 generic.go:356] "Generic (PLEG): container finished" podID="153210bf-399a-41bd-a5e3-de2a38b7d549" containerID="dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241" exitCode=0
Oct 30 17:20:07.672768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.672284    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"
Oct 30 17:20:07.672768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.672324    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6" event={"ID":"153210bf-399a-41bd-a5e3-de2a38b7d549","Type":"ContainerDied","Data":"dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241"}
Oct 30 17:20:07.672768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.672381    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6" event={"ID":"153210bf-399a-41bd-a5e3-de2a38b7d549","Type":"ContainerDied","Data":"ccc8ee83cf6a1f4f009c43594fe9f67e612a73586530afdeb7219b5a0829565c"}
Oct 30 17:20:07.672768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.672403    4497 scope.go:122] "RemoveContainer" containerID="dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241"
Oct 30 17:20:07.689118 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.689099    4497 scope.go:122] "RemoveContainer" containerID="dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241"
Oct 30 17:20:07.689402 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:20:07.689381    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241\": container with ID starting with dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241 not found: ID does not exist" containerID="dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241"
Oct 30 17:20:07.689481 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.689413    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241"} err="failed to get container status \"dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241\": rpc error: code = NotFound desc = could not find container \"dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241\": container with ID starting with dc1c43622ecd9ebf5724ab3bbfd5a29d94514943a5b565d27a93456238665241 not found: ID does not exist"
Oct 30 17:20:07.720502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.720480    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"]
Oct 30 17:20:07.729772 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:07.729751    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6acc78-76865d574-njzw6"]
Oct 30 17:20:08.301874 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:08.301843    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153210bf-399a-41bd-a5e3-de2a38b7d549" path="/var/lib/kubelet/pods/153210bf-399a-41bd-a5e3-de2a38b7d549/volumes"
Oct 30 17:20:17.040517 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:17.040482    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"]
Oct 30 17:20:17.048932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:17.048906    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353da911-kh97t"]
Oct 30 17:20:18.302410 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:18.302366    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7459f5d-7db1-4bea-a17f-4a6d61ef3495" path="/var/lib/kubelet/pods/e7459f5d-7db1-4bea-a17f-4a6d61ef3495/volumes"
Oct 30 17:20:21.917096 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:20:21.917061    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:20:25.978404 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:25.978369    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"]
Oct 30 17:20:30.978863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:30.978817    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d" podUID="1151b05d-9e88-46f5-9e73-2c7b7b8e8829" containerName="payload-processing" containerID="cri-o://f3a4ab434bf065b760588c12bc963f8461f256205115f95b15fc1e3c090fb9b2" gracePeriod=25
Oct 30 17:20:33.072168 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:33.072087    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d" podUID="1151b05d-9e88-46f5-9e73-2c7b7b8e8829" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:20:43.072438 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:43.072393    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d" podUID="1151b05d-9e88-46f5-9e73-2c7b7b8e8829" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:20:51.276216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:51.276167    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:20:51.305179 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:51.305153    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-kube-api-access-xm5ss\" (UniqueName: \"kubernetes.io/projected/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-kube-api-access-xm5ss\") pod \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\" (UID: \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\") "
Oct 30 17:20:51.305316 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:51.305300    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-plugins-config-volume\") pod \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\" (UID: \"1151b05d-9e88-46f5-9e73-2c7b7b8e8829\") "
Oct 30 17:20:51.305704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:51.305677    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-plugins-config-volume" pod "1151b05d-9e88-46f5-9e73-2c7b7b8e8829" (UID: "1151b05d-9e88-46f5-9e73-2c7b7b8e8829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:20:51.305935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:51.305914    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:20:51.307393 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:51.307369    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-kube-api-access-xm5ss" pod "1151b05d-9e88-46f5-9e73-2c7b7b8e8829" (UID: "1151b05d-9e88-46f5-9e73-2c7b7b8e8829"). InnerVolumeSpecName "kube-api-access-xm5ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:20:51.407532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:51.407501    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5ss\" (UniqueName: \"kubernetes.io/projected/1151b05d-9e88-46f5-9e73-2c7b7b8e8829-kube-api-access-xm5ss\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:20:51.905610 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:51.905583    4497 scope.go:122] "RemoveContainer" containerID="f3a4ab434bf065b760588c12bc963f8461f256205115f95b15fc1e3c090fb9b2"
Oct 30 17:20:51.917131 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:20:51.917095    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:20:52.064813 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:52.064777    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d" event={"ID":"1151b05d-9e88-46f5-9e73-2c7b7b8e8829","Type":"ContainerDied","Data":"f3a4ab434bf065b760588c12bc963f8461f256205115f95b15fc1e3c090fb9b2"}
Oct 30 17:20:52.064989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:52.064852    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d" event={"ID":"1151b05d-9e88-46f5-9e73-2c7b7b8e8829","Type":"ContainerDied","Data":"ea84552cca805f3f2ae10a6f4d578e1c073d733e2ee1458f271236dbdab9d65e"}
Oct 30 17:20:52.064989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:52.064867    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"
Oct 30 17:20:52.135603 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:52.135572    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"]
Oct 30 17:20:52.141839 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:52.141815    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24acb2-76c4447b66-h622d"]
Oct 30 17:20:52.301925 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:20:52.301892    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1151b05d-9e88-46f5-9e73-2c7b7b8e8829" path="/var/lib/kubelet/pods/1151b05d-9e88-46f5-9e73-2c7b7b8e8829/volumes"
Oct 30 17:21:15.502181 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:15.502147    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"]
Oct 30 17:21:15.502689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:15.502359    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy" containerID="cri-o://92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a" gracePeriod=30
Oct 30 17:21:20.284771 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.284739    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"]
Oct 30 17:21:20.285252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.284961    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd" podUID="48b6e4b9-63df-4089-86b2-7fdb181c1e78" containerName="istio-proxy" containerID="cri-o://c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7" gracePeriod=30
Oct 30 17:21:20.792714 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.792684    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 17:21:20.969536 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969455    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-data\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.969706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969540    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-envoy\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.969706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969600    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-socket\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.969706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969637    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-kube-api-access-r467s\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-kube-api-access-r467s\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.969706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969693    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-token\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-token\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.969957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969784    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-data" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:20.969957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969888    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-certs\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.969957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969892    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-socket" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:20.969957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.969928    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-credential-socket\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.970173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970003    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0fca89-11f6-4450-9b7b-27f9fe014668-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0fca89-11f6-4450-9b7b-27f9fe014668-istiod-ca-cert\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.970173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970108    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-certs" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:20.970173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970127    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-podinfo\") pod \"0a0fca89-11f6-4450-9b7b-27f9fe014668\" (UID: \"0a0fca89-11f6-4450-9b7b-27f9fe014668\") "
Oct 30 17:21:20.970366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970167    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-credential-socket" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:20.970429 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970414    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0fca89-11f6-4450-9b7b-27f9fe014668-istiod-ca-cert" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:20.970924 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970901    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:20.971031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970924    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:20.971031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970939    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:20.971031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970950    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a0fca89-11f6-4450-9b7b-27f9fe014668-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:20.971031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.970962    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:20.971887 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.971862    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-envoy" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:20.972105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.972086    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-podinfo" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:21:20.972328 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.972312    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-kube-api-access-r467s" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "kube-api-access-r467s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:20.972405 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:20.972336    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-token" pod "0a0fca89-11f6-4450-9b7b-27f9fe014668" (UID: "0a0fca89-11f6-4450-9b7b-27f9fe014668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:21.072013 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.071988    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r467s\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-kube-api-access-r467s\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:21.072013 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.072012    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:21.072198 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.072021    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:21.072198 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.072029    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a0fca89-11f6-4450-9b7b-27f9fe014668-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:21.315317 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.315283    4497 generic.go:356] "Generic (PLEG): container finished" podID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerID="92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a" exitCode=0
Oct 30 17:21:21.315805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.315340    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"
Oct 30 17:21:21.315805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.315359    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" event={"ID":"0a0fca89-11f6-4450-9b7b-27f9fe014668","Type":"ContainerDied","Data":"92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a"}
Oct 30 17:21:21.315805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.315442    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q" event={"ID":"0a0fca89-11f6-4450-9b7b-27f9fe014668","Type":"ContainerDied","Data":"122538d0e3084161941dd904b7222b6da8dbcca76f9a4a4cef86a2a87ed63420"}
Oct 30 17:21:21.315805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.315470    4497 scope.go:122] "RemoveContainer" containerID="92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a"
Oct 30 17:21:21.326885 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.326867    4497 scope.go:122] "RemoveContainer" containerID="f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b"
Oct 30 17:21:21.338097 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.338082    4497 scope.go:122] "RemoveContainer" containerID="92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a"
Oct 30 17:21:21.338358 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:21.338339    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a\": container with ID starting with 92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a not found: ID does not exist" containerID="92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a"
Oct 30 17:21:21.338402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.338367    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a"} err="failed to get container status \"92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a\": rpc error: code = NotFound desc = could not find container \"92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a\": container with ID starting with 92f0652b3c8848a8b48d214054f8308459eb8198deefb2a049c132ebb28bac4a not found: ID does not exist"
Oct 30 17:21:21.338402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.338383    4497 scope.go:122] "RemoveContainer" containerID="f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b"
Oct 30 17:21:21.338620 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:21.338607    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b\": container with ID starting with f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b not found: ID does not exist" containerID="f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b"
Oct 30 17:21:21.338661 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.338624    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b"} err="failed to get container status \"f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b\": rpc error: code = NotFound desc = could not find container \"f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b\": container with ID starting with f9b2542e7f73d39206269ade50532820e7b96af4648a3eb36fbd56d9fe87bd3b not found: ID does not exist"
Oct 30 17:21:21.364438 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.364410    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"]
Oct 30 17:21:21.371847 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.371826    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-7gs7q"]
Oct 30 17:21:21.709490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.709404    4497 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:21:21.709490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:21.709455    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd" podUID="48b6e4b9-63df-4089-86b2-7fdb181c1e78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:21:21.916898 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:21.916864    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:21:22.304794 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:22.304758    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" path="/var/lib/kubelet/pods/0a0fca89-11f6-4450-9b7b-27f9fe014668/volumes"
Oct 30 17:21:25.583193 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.583159    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:21:25.635041 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635014    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-certs\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635066    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-token\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-token\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635133    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-socket\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635329 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635169    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-kube-api-access-ltsqd\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-kube-api-access-ltsqd\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635329 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635228    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-envoy\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635435 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635343    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-data\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635435 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635426    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-podinfo\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635547 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635482    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-certs" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:25.635547 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635534    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-credential-socket\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635569    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istiod-ca-cert\") pod \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\" (UID: \"48b6e4b9-63df-4089-86b2-7fdb181c1e78\") "
Oct 30 17:21:25.635825 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.635798    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-data" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:25.636150 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.636128    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istiod-ca-cert" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:25.636595 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.636574    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:25.636733 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.636721    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:25.636837 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.636826    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:25.636914 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.636677    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-socket" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:25.637004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.636823    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-credential-socket" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:25.638222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.638171    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-token" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:25.638816 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.638795    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-envoy" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:25.640011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.639976    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-kube-api-access-ltsqd" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "kube-api-access-ltsqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:25.640336 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.640313    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-podinfo" pod "48b6e4b9-63df-4089-86b2-7fdb181c1e78" (UID: "48b6e4b9-63df-4089-86b2-7fdb181c1e78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:21:25.738176 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.738143    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:25.738176 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.738169    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:25.738439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.738180    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:25.738439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.738224    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:25.738439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.738233    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b6e4b9-63df-4089-86b2-7fdb181c1e78-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:25.738439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:25.738241    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltsqd\" (UniqueName: \"kubernetes.io/projected/48b6e4b9-63df-4089-86b2-7fdb181c1e78-kube-api-access-ltsqd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:26.354695 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.354649    4497 generic.go:356] "Generic (PLEG): container finished" podID="48b6e4b9-63df-4089-86b2-7fdb181c1e78" containerID="c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7" exitCode=0
Oct 30 17:21:26.354695 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.354696    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"
Oct 30 17:21:26.354964 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.354694    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd" event={"ID":"48b6e4b9-63df-4089-86b2-7fdb181c1e78","Type":"ContainerDied","Data":"c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7"}
Oct 30 17:21:26.354964 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.354797    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd" event={"ID":"48b6e4b9-63df-4089-86b2-7fdb181c1e78","Type":"ContainerDied","Data":"3b2e0b2e064da27bca7a7ad4845f7d5bed05323b66abf6d79b369313c741c22d"}
Oct 30 17:21:26.354964 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.354830    4497 scope.go:122] "RemoveContainer" containerID="c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7"
Oct 30 17:21:26.365552 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.365529    4497 scope.go:122] "RemoveContainer" containerID="c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7"
Oct 30 17:21:26.365833 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:26.365805    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7\": container with ID starting with c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7 not found: ID does not exist" containerID="c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7"
Oct 30 17:21:26.365892 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.365838    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7"} err="failed to get container status \"c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7\": rpc error: code = NotFound desc = could not find container \"c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7\": container with ID starting with c1bc4d155bc023fa0353f926359cd83b0dd11c592f4d632877c29b9e043da9f7 not found: ID does not exist"
Oct 30 17:21:26.413915 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.413876    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"]
Oct 30 17:21:26.423217 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:26.423178    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-c5rdd"]
Oct 30 17:21:27.820532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:27.820486    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"]
Oct 30 17:21:27.821019 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:27.820702    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy" containerID="cri-o://9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d" gracePeriod=30
Oct 30 17:21:28.303104 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:28.303071    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b6e4b9-63df-4089-86b2-7fdb181c1e78" path="/var/lib/kubelet/pods/48b6e4b9-63df-4089-86b2-7fdb181c1e78/volumes"
Oct 30 17:21:30.574132 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:30.574100    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"]
Oct 30 17:21:30.574549 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:30.574306    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm" podUID="157e4427-efba-47e6-8332-f60180eca846" containerName="discovery" containerID="cri-o://d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf" gracePeriod=30
Oct 30 17:21:30.871149 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:30.871120    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:21:31.024114 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.022458    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157e4427-efba-47e6-8332-f60180eca846-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/157e4427-efba-47e6-8332-f60180eca846-istio-csr-ca-configmap\") pod \"157e4427-efba-47e6-8332-f60180eca846\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") "
Oct 30 17:21:31.024114 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.022625    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-cacerts\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-cacerts\") pod \"157e4427-efba-47e6-8332-f60180eca846\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") "
Oct 30 17:21:31.024114 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.022704    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157e4427-efba-47e6-8332-f60180eca846-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/157e4427-efba-47e6-8332-f60180eca846-local-certs\") pod \"157e4427-efba-47e6-8332-f60180eca846\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") "
Oct 30 17:21:31.024114 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.022760    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-kube-api-access-ptxgc\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-kube-api-access-ptxgc\") pod \"157e4427-efba-47e6-8332-f60180eca846\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") "
Oct 30 17:21:31.024114 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.022891    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-csr-dns-cert\") pod \"157e4427-efba-47e6-8332-f60180eca846\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") "
Oct 30 17:21:31.024114 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.022986    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-istio-token\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-istio-token\") pod \"157e4427-efba-47e6-8332-f60180eca846\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") "
Oct 30 17:21:31.024114 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.023067    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-kubeconfig\") pod \"157e4427-efba-47e6-8332-f60180eca846\" (UID: \"157e4427-efba-47e6-8332-f60180eca846\") "
Oct 30 17:21:31.025334 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.025020    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157e4427-efba-47e6-8332-f60180eca846-istio-csr-ca-configmap" pod "157e4427-efba-47e6-8332-f60180eca846" (UID: "157e4427-efba-47e6-8332-f60180eca846"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:31.029772 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.029675    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-kubeconfig" pod "157e4427-efba-47e6-8332-f60180eca846" (UID: "157e4427-efba-47e6-8332-f60180eca846"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:21:31.029772 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.029701    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-cacerts" pod "157e4427-efba-47e6-8332-f60180eca846" (UID: "157e4427-efba-47e6-8332-f60180eca846"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:21:31.029966 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.029786    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157e4427-efba-47e6-8332-f60180eca846-local-certs" pod "157e4427-efba-47e6-8332-f60180eca846" (UID: "157e4427-efba-47e6-8332-f60180eca846"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:31.029966 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.029861    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-kube-api-access-ptxgc" pod "157e4427-efba-47e6-8332-f60180eca846" (UID: "157e4427-efba-47e6-8332-f60180eca846"). InnerVolumeSpecName "kube-api-access-ptxgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:31.030894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.030872    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-istio-token" pod "157e4427-efba-47e6-8332-f60180eca846" (UID: "157e4427-efba-47e6-8332-f60180eca846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:31.031138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.031120    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-csr-dns-cert" pod "157e4427-efba-47e6-8332-f60180eca846" (UID: "157e4427-efba-47e6-8332-f60180eca846"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:21:31.125713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.125650    4497 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-cacerts\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:31.125713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.125672    4497 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/157e4427-efba-47e6-8332-f60180eca846-local-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:31.125713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.125682    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxgc\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-kube-api-access-ptxgc\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:31.125713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.125691    4497 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-csr-dns-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:31.125713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.125699    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157e4427-efba-47e6-8332-f60180eca846-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:31.125713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.125707    4497 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/157e4427-efba-47e6-8332-f60180eca846-istio-kubeconfig\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:31.125713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.125714    4497 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/157e4427-efba-47e6-8332-f60180eca846-istio-csr-ca-configmap\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:31.394902 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.394816    4497 generic.go:356] "Generic (PLEG): container finished" podID="157e4427-efba-47e6-8332-f60180eca846" containerID="d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf" exitCode=0
Oct 30 17:21:31.394902 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.394865    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"
Oct 30 17:21:31.395087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.394910    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm" event={"ID":"157e4427-efba-47e6-8332-f60180eca846","Type":"ContainerDied","Data":"d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf"}
Oct 30 17:21:31.395087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.394956    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm" event={"ID":"157e4427-efba-47e6-8332-f60180eca846","Type":"ContainerDied","Data":"7922b726efc401b909b4d65bbf465e41f02d06578e0344ab583cd0a9d56ea78f"}
Oct 30 17:21:31.395087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.394976    4497 scope.go:122] "RemoveContainer" containerID="d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf"
Oct 30 17:21:31.407065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.407042    4497 scope.go:122] "RemoveContainer" containerID="d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf"
Oct 30 17:21:31.407366 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:31.407347    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf\": container with ID starting with d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf not found: ID does not exist" containerID="d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf"
Oct 30 17:21:31.407421 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.407375    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf"} err="failed to get container status \"d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf\": rpc error: code = NotFound desc = could not find container \"d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf\": container with ID starting with d0bb879128bb16242b80f80d1a034e623e6b2aafcdfc02cf5d072fa8796222cf not found: ID does not exist"
Oct 30 17:21:31.445554 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.445532    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"]
Oct 30 17:21:31.454365 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:31.454345    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zbvxm"]
Oct 30 17:21:32.301623 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:32.301588    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157e4427-efba-47e6-8332-f60180eca846" path="/var/lib/kubelet/pods/157e4427-efba-47e6-8332-f60180eca846/volumes"
Oct 30 17:21:33.104130 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.104076    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:21:33.260052 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260022    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-istio-token\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-istio-token\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260256 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260119    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3de812-692e-405a-953c-cc84de94107a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3de812-692e-405a-953c-cc84de94107a-istiod-ca-cert\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260256 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260142    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-socket\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260256 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260162    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-kube-api-access-6jjlg\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-kube-api-access-6jjlg\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260256 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260222    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-certs\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260536 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260512    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-socket" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:33.260610 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260564    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-data\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260610 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260582    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-certs" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:33.260610 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260603    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-credential-socket\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260610 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260588    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3de812-692e-405a-953c-cc84de94107a-istiod-ca-cert" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:33.260807 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260661    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e3de812-692e-405a-953c-cc84de94107a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3de812-692e-405a-953c-cc84de94107a-istio-podinfo\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260807 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260698    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-envoy\") pod \"7e3de812-692e-405a-953c-cc84de94107a\" (UID: \"7e3de812-692e-405a-953c-cc84de94107a\") "
Oct 30 17:21:33.260927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260830    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-data" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:33.260927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.260856    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-credential-socket" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:33.261425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.261341    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.261425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.261366    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.261425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.261378    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3de812-692e-405a-953c-cc84de94107a-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.261425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.261387    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.261425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.261395    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.262784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.262753    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-istio-token" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:33.262923 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.262891    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-kube-api-access-6jjlg" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "kube-api-access-6jjlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:33.263052 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.263029    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e3de812-692e-405a-953c-cc84de94107a-istio-podinfo" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:21:33.263117 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.263107    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-envoy" pod "7e3de812-692e-405a-953c-cc84de94107a" (UID: "7e3de812-692e-405a-953c-cc84de94107a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:33.363106 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.363023    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3de812-692e-405a-953c-cc84de94107a-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.363106 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.363060    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3de812-692e-405a-953c-cc84de94107a-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.363106 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.363073    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.363106 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.363086    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jjlg\" (UniqueName: \"kubernetes.io/projected/7e3de812-692e-405a-953c-cc84de94107a-kube-api-access-6jjlg\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:33.414277 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.414244    4497 generic.go:356] "Generic (PLEG): container finished" podID="7e3de812-692e-405a-953c-cc84de94107a" containerID="9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d" exitCode=0
Oct 30 17:21:33.414440 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.414292    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"
Oct 30 17:21:33.414440 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.414327    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" event={"ID":"7e3de812-692e-405a-953c-cc84de94107a","Type":"ContainerDied","Data":"9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d"}
Oct 30 17:21:33.414440 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.414397    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng" event={"ID":"7e3de812-692e-405a-953c-cc84de94107a","Type":"ContainerDied","Data":"ef3312599949a2e6fabc7085189620a0b78b0d38f3fbfd8202b96e3247fd0ce6"}
Oct 30 17:21:33.414440 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.414429    4497 scope.go:122] "RemoveContainer" containerID="9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d"
Oct 30 17:21:33.428815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.428714    4497 scope.go:122] "RemoveContainer" containerID="9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d"
Oct 30 17:21:33.429063 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:33.429036    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d\": container with ID starting with 9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d not found: ID does not exist" containerID="9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d"
Oct 30 17:21:33.429063 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.429079    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d"} err="failed to get container status \"9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d\": rpc error: code = NotFound desc = could not find container \"9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d\": container with ID starting with 9611afa4a4903890de5a8014319d1a7c33440f0f89dcc88337c77a32fc043c1d not found: ID does not exist"
Oct 30 17:21:33.462394 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.462365    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"]
Oct 30 17:21:33.467028 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:33.467003    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-9pqng"]
Oct 30 17:21:34.103402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:34.103374    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"]
Oct 30 17:21:34.103621 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:34.103593    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy" containerID="cri-o://07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23" gracePeriod=30
Oct 30 17:21:34.301677 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:34.301645    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3de812-692e-405a-953c-cc84de94107a" path="/var/lib/kubelet/pods/7e3de812-692e-405a-953c-cc84de94107a/volumes"
Oct 30 17:21:38.923341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:38.923305    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"]
Oct 30 17:21:39.385879 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.385855    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:21:39.450037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.450003    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-istio-token\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-istio-token\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.450547 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.450509    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-credential-socket\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.450671 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.450604    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-certs\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.450776 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.450758    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-kube-api-access-t85tn\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-kube-api-access-t85tn\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.450844 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.450809    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c597a45-d66f-4231-8243-4867e0850f31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c597a45-d66f-4231-8243-4867e0850f31-istiod-ca-cert\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.450905 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.450849    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-envoy\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.451021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.450997    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c597a45-d66f-4231-8243-4867e0850f31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c597a45-d66f-4231-8243-4867e0850f31-istio-podinfo\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.451131 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.451048    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-socket\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.451131 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.451095    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-data\") pod \"3c597a45-d66f-4231-8243-4867e0850f31\" (UID: \"3c597a45-d66f-4231-8243-4867e0850f31\") "
Oct 30 17:21:39.452059 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.452029    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-data" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.452266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.452245    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-credential-socket" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.452453 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.452433    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-certs" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.453172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.453145    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-istio-token" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:39.453546 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.453522    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c597a45-d66f-4231-8243-4867e0850f31-istiod-ca-cert" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:39.454855 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.454827    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-kube-api-access-t85tn" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "kube-api-access-t85tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:39.454929 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.454831    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-envoy" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.455035 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.455014    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-socket" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:39.455741 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.455712    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c597a45-d66f-4231-8243-4867e0850f31-istio-podinfo" pod "3c597a45-d66f-4231-8243-4867e0850f31" (UID: "3c597a45-d66f-4231-8243-4867e0850f31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:21:39.467889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.467864    4497 generic.go:356] "Generic (PLEG): container finished" podID="3c597a45-d66f-4231-8243-4867e0850f31" containerID="07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23" exitCode=0
Oct 30 17:21:39.467984 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.467935    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"
Oct 30 17:21:39.467984 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.467945    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" event={"ID":"3c597a45-d66f-4231-8243-4867e0850f31","Type":"ContainerDied","Data":"07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23"}
Oct 30 17:21:39.468098 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.468001    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r" event={"ID":"3c597a45-d66f-4231-8243-4867e0850f31","Type":"ContainerDied","Data":"5cd4e3462e11aa04ce6c5e94524faaf36ba7dcc023aa8acc479e7fc06e026b75"}
Oct 30 17:21:39.468098 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.468029    4497 scope.go:122] "RemoveContainer" containerID="07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23"
Oct 30 17:21:39.498550 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.498534    4497 scope.go:122] "RemoveContainer" containerID="07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23"
Oct 30 17:21:39.498804 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:39.498780    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23\": container with ID starting with 07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23 not found: ID does not exist" containerID="07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23"
Oct 30 17:21:39.498895 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.498808    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23"} err="failed to get container status \"07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23\": rpc error: code = NotFound desc = could not find container \"07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23\": container with ID starting with 07bb7bd9aa2d4207ade20ab81ac167392dc1e238f7a0d3d6432e6b23feeffd23 not found: ID does not exist"
Oct 30 17:21:39.527549 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.527527    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"]
Oct 30 17:21:39.531837 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.531815    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-ct79r"]
Oct 30 17:21:39.553387 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553366    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.553387 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553386    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.553508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553395    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.553508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553405    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t85tn\" (UniqueName: \"kubernetes.io/projected/3c597a45-d66f-4231-8243-4867e0850f31-kube-api-access-t85tn\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.553508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553416    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c597a45-d66f-4231-8243-4867e0850f31-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.553508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553423    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.553508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553431    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c597a45-d66f-4231-8243-4867e0850f31-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.553508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553439    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:39.553508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:39.553447    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c597a45-d66f-4231-8243-4867e0850f31-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:40.304558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:40.304521    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c597a45-d66f-4231-8243-4867e0850f31" path="/var/lib/kubelet/pods/3c597a45-d66f-4231-8243-4867e0850f31/volumes"
Oct 30 17:21:41.121079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:41.121043    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"]
Oct 30 17:21:41.121303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:41.121247    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t" podUID="3df30ece-9a53-4a35-be67-d8e6675d6ba8" containerName="istio-proxy" containerID="cri-o://bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492" gracePeriod=30
Oct 30 17:21:42.165561 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:42.165524    4497 patch_prober.go:29] interesting pod/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:21:42.166017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:42.165587    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t" podUID="3df30ece-9a53-4a35-be67-d8e6675d6ba8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:21:43.924054 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:43.924018    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77" podUID="5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" containerName="payload-pre-processing" containerID="cri-o://7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4" gracePeriod=25
Oct 30 17:21:46.447068 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.447036    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:21:46.524121 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.524082    4497 generic.go:356] "Generic (PLEG): container finished" podID="3df30ece-9a53-4a35-be67-d8e6675d6ba8" containerID="bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492" exitCode=0
Oct 30 17:21:46.524481 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.524166    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t" event={"ID":"3df30ece-9a53-4a35-be67-d8e6675d6ba8","Type":"ContainerDied","Data":"bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492"}
Oct 30 17:21:46.524481 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.524283    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t" event={"ID":"3df30ece-9a53-4a35-be67-d8e6675d6ba8","Type":"ContainerDied","Data":"7c6bf97958c41447f107cbe9266502aedc130778688557070d42a38106dd8edc"}
Oct 30 17:21:46.524481 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.524317    4497 scope.go:122] "RemoveContainer" containerID="bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492"
Oct 30 17:21:46.524481 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.524463    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"
Oct 30 17:21:46.535942 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.535800    4497 scope.go:122] "RemoveContainer" containerID="bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492"
Oct 30 17:21:46.536299 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:46.536217    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492\": container with ID starting with bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492 not found: ID does not exist" containerID="bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492"
Oct 30 17:21:46.536299 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.536252    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492"} err="failed to get container status \"bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492\": rpc error: code = NotFound desc = could not find container \"bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492\": container with ID starting with bf2a4384b6c09172683902f0a173cb1e69536a50f61b75c1d904f923a9bdf492 not found: ID does not exist"
Oct 30 17:21:46.568293 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568264    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-certs\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.568403 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568349    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-token\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-token\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.568403 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568373    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-podinfo\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.568561 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568536    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-credential-socket\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.568629 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568581    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-certs" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:46.568692 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568637    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-kube-api-access-75j6d\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-kube-api-access-75j6d\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.568745 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568696    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-envoy\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.568795 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568768    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istiod-ca-cert\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.568860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.568830    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-credential-socket" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:46.569232 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.569163    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istiod-ca-cert" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:46.569492 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.569462    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-socket\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.569697 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.569524    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-data\") pod \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\" (UID: \"3df30ece-9a53-4a35-be67-d8e6675d6ba8\") "
Oct 30 17:21:46.569786 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.569747    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-socket" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:46.569853 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.569789    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-data" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:46.570420 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.570390    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.570527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.570424    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.570527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.570442    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.570527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.570456    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.570527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.570469    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.570805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.570786    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-token" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:46.570920 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.570895    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-podinfo" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:21:46.571042 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.570953    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-kube-api-access-75j6d" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "kube-api-access-75j6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:46.571042 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.571030    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-envoy" pod "3df30ece-9a53-4a35-be67-d8e6675d6ba8" (UID: "3df30ece-9a53-4a35-be67-d8e6675d6ba8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:46.672567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.672488    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75j6d\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-kube-api-access-75j6d\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.672567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.672521    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.672567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.672535    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.672567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.672547    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df30ece-9a53-4a35-be67-d8e6675d6ba8-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:46.867629 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.867603    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"]
Oct 30 17:21:46.876924 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:46.876897    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789prr7t"]
Oct 30 17:21:48.301200 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:48.301155    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df30ece-9a53-4a35-be67-d8e6675d6ba8" path="/var/lib/kubelet/pods/3df30ece-9a53-4a35-be67-d8e6675d6ba8/volumes"
Oct 30 17:21:51.916685 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:51.916652    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:21:51.938787 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:51.938759    4497 scope.go:122] "RemoveContainer" containerID="2f68ed6189e629c49783453a897fdf2c47ddc053384510505a425abbb305638a"
Oct 30 17:21:52.201285 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:52.201211    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77" podUID="5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:21:53.935860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:53.935821    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"]
Oct 30 17:21:53.936539 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:53.936104    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy" containerID="cri-o://5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b" gracePeriod=30
Oct 30 17:21:58.400820 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:58.400788    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:21:58.401246 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:58.400849    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:21:59.230640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.230609    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 17:21:59.347632 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.347596    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-kube-api-access-k9kz8\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-kube-api-access-k9kz8\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.347806 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.347719    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-envoy\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.347933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.347910    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-socket\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.348021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348005    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-data\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.348101 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348040    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9905c187-904a-44ee-ab84-f6e767fc887b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905c187-904a-44ee-ab84-f6e767fc887b-istio-podinfo\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.348101 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348083    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9905c187-904a-44ee-ab84-f6e767fc887b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905c187-904a-44ee-ab84-f6e767fc887b-istiod-ca-cert\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.348221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348113    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-socket" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:59.348221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348132    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-credential-socket\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.348337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348241    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-istio-token\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-istio-token\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.348337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348290    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-certs\") pod \"9905c187-904a-44ee-ab84-f6e767fc887b\" (UID: \"9905c187-904a-44ee-ab84-f6e767fc887b\") "
Oct 30 17:21:59.348458 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348344    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-data" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:59.348458 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348359    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-credential-socket" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:59.348623 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348603    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-certs" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:59.348801 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.348776    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9905c187-904a-44ee-ab84-f6e767fc887b-istiod-ca-cert" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:21:59.349150 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.349131    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.349150 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.349150    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.349357 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.349158    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.349357 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.349167    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905c187-904a-44ee-ab84-f6e767fc887b-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.349357 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.349180    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.350077 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.350052    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-envoy" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:21:59.350236 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.350213    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-kube-api-access-k9kz8" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "kube-api-access-k9kz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:59.350611 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.350583    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9905c187-904a-44ee-ab84-f6e767fc887b-istio-podinfo" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:21:59.350705 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.350618    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-istio-token" pod "9905c187-904a-44ee-ab84-f6e767fc887b" (UID: "9905c187-904a-44ee-ab84-f6e767fc887b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:21:59.450922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.450886    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.450922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.450915    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9kz8\" (UniqueName: \"kubernetes.io/projected/9905c187-904a-44ee-ab84-f6e767fc887b-kube-api-access-k9kz8\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.450922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.450925    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905c187-904a-44ee-ab84-f6e767fc887b-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.451426 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.450934    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905c187-904a-44ee-ab84-f6e767fc887b-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:21:59.617725 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.617686    4497 generic.go:356] "Generic (PLEG): container finished" podID="9905c187-904a-44ee-ab84-f6e767fc887b" containerID="5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b" exitCode=0
Oct 30 17:21:59.617911 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.617742    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"
Oct 30 17:21:59.617911 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.617763    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" event={"ID":"9905c187-904a-44ee-ab84-f6e767fc887b","Type":"ContainerDied","Data":"5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b"}
Oct 30 17:21:59.617911 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.617822    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t" event={"ID":"9905c187-904a-44ee-ab84-f6e767fc887b","Type":"ContainerDied","Data":"07dd5048f59bd59b2467993b876732b8f8ac3b0b2d305e93eaf7c07b57e99415"}
Oct 30 17:21:59.617911 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.617852    4497 scope.go:122] "RemoveContainer" containerID="5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b"
Oct 30 17:21:59.629685 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.629657    4497 scope.go:122] "RemoveContainer" containerID="33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94"
Oct 30 17:21:59.639571 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.639552    4497 scope.go:122] "RemoveContainer" containerID="5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b"
Oct 30 17:21:59.639899 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:59.639875    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b\": container with ID starting with 5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b not found: ID does not exist" containerID="5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b"
Oct 30 17:21:59.639963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.639911    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b"} err="failed to get container status \"5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b\": rpc error: code = NotFound desc = could not find container \"5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b\": container with ID starting with 5917765a12946a7da41c1fdc5f44c6cb65113a56bb1c877f9b7d854b14aa2d9b not found: ID does not exist"
Oct 30 17:21:59.639963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.639935    4497 scope.go:122] "RemoveContainer" containerID="33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94"
Oct 30 17:21:59.640248 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:21:59.640227    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94\": container with ID starting with 33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94 not found: ID does not exist" containerID="33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94"
Oct 30 17:21:59.640312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.640257    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94"} err="failed to get container status \"33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94\": rpc error: code = NotFound desc = could not find container \"33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94\": container with ID starting with 33fc748fb89f50c414c1798360a0db409143e6adf0aa06b836d54bf7788b3c94 not found: ID does not exist"
Oct 30 17:21:59.665160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.665132    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"]
Oct 30 17:21:59.675028 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:21:59.675000    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-5dj2t"]
Oct 30 17:22:00.301391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:00.301354    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" path="/var/lib/kubelet/pods/9905c187-904a-44ee-ab84-f6e767fc887b/volumes"
Oct 30 17:22:02.201517 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:02.201481    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77" podUID="5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:22:04.210820 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.210792    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 17:22:04.323742 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.323716    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-kube-api-access-99q2z\" (UniqueName: \"kubernetes.io/projected/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-kube-api-access-99q2z\") pod \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\" (UID: \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\") "
Oct 30 17:22:04.323952 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.323924    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-plugins-config-volume\") pod \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\" (UID: \"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51\") "
Oct 30 17:22:04.324395 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.324367    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-plugins-config-volume" pod "5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" (UID: "5cee62a7-517b-4a8b-8d59-0be8dd2c6d51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:22:04.325763 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.325742    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-kube-api-access-99q2z" pod "5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" (UID: "5cee62a7-517b-4a8b-8d59-0be8dd2c6d51"). InnerVolumeSpecName "kube-api-access-99q2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:04.426564 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.426525    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:22:04.426564 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.426557    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99q2z\" (UniqueName: \"kubernetes.io/projected/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51-kube-api-access-99q2z\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:22:04.656917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.656883    4497 generic.go:356] "Generic (PLEG): container finished" podID="5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" containerID="7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4" exitCode=0
Oct 30 17:22:04.657086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.656940    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"
Oct 30 17:22:04.657086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.656961    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77" event={"ID":"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51","Type":"ContainerDied","Data":"7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4"}
Oct 30 17:22:04.657086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.657047    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77" event={"ID":"5cee62a7-517b-4a8b-8d59-0be8dd2c6d51","Type":"ContainerDied","Data":"fa3016e73ce60650163c291d0c61189da009d2edb7d465478e78a971482d3859"}
Oct 30 17:22:04.657086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.657077    4497 scope.go:122] "RemoveContainer" containerID="7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4"
Oct 30 17:22:04.668796 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.668779    4497 scope.go:122] "RemoveContainer" containerID="7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4"
Oct 30 17:22:04.669081 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:22:04.669054    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4\": container with ID starting with 7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4 not found: ID does not exist" containerID="7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4"
Oct 30 17:22:04.669160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.669092    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4"} err="failed to get container status \"7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4\": rpc error: code = NotFound desc = could not find container \"7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4\": container with ID starting with 7d2514086045b2d630aae88c7bcffcb760ab890dda5e1f1a57b84338805fa2d4 not found: ID does not exist"
Oct 30 17:22:04.701997 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.701966    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"]
Oct 30 17:22:04.705474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:04.705453    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d5b798-75b6c774c5-chs77"]
Oct 30 17:22:05.032815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:05.032785    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"]
Oct 30 17:22:06.303888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:06.303854    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" path="/var/lib/kubelet/pods/5cee62a7-517b-4a8b-8d59-0be8dd2c6d51/volumes"
Oct 30 17:22:10.033748 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:10.033706    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf" podUID="a124b7a1-5f18-4779-a839-3c7a8c37c5f8" containerName="payload-pre-processing" containerID="cri-o://16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0" gracePeriod=25
Oct 30 17:22:18.929821 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:18.929723    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf" podUID="a124b7a1-5f18-4779-a839-3c7a8c37c5f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:22:21.917046 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:22:21.917011    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:22:28.945841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:28.945780    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf" podUID="a124b7a1-5f18-4779-a839-3c7a8c37c5f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:22:30.320708 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.320676    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:22:30.400441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.400411    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-plugins-config-volume\") pod \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\" (UID: \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\") "
Oct 30 17:22:30.400441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.400448    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-kube-api-access-b2p46\" (UniqueName: \"kubernetes.io/projected/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-kube-api-access-b2p46\") pod \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\" (UID: \"a124b7a1-5f18-4779-a839-3c7a8c37c5f8\") "
Oct 30 17:22:30.400788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.400759    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-plugins-config-volume" pod "a124b7a1-5f18-4779-a839-3c7a8c37c5f8" (UID: "a124b7a1-5f18-4779-a839-3c7a8c37c5f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:22:30.401011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.400991    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:22:30.402744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.402720    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-kube-api-access-b2p46" pod "a124b7a1-5f18-4779-a839-3c7a8c37c5f8" (UID: "a124b7a1-5f18-4779-a839-3c7a8c37c5f8"). InnerVolumeSpecName "kube-api-access-b2p46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:30.501878 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.501840    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2p46\" (UniqueName: \"kubernetes.io/projected/a124b7a1-5f18-4779-a839-3c7a8c37c5f8-kube-api-access-b2p46\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:22:30.861146 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.861105    4497 generic.go:356] "Generic (PLEG): container finished" podID="a124b7a1-5f18-4779-a839-3c7a8c37c5f8" containerID="16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0" exitCode=0
Oct 30 17:22:30.861360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.861149    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"
Oct 30 17:22:30.861360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.861223    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf" event={"ID":"a124b7a1-5f18-4779-a839-3c7a8c37c5f8","Type":"ContainerDied","Data":"16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0"}
Oct 30 17:22:30.861360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.861279    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf" event={"ID":"a124b7a1-5f18-4779-a839-3c7a8c37c5f8","Type":"ContainerDied","Data":"54abfbb84b262e8eb878fcce50bd4f70fde7ada42f06c70d734dbf563075d8df"}
Oct 30 17:22:30.861360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.861306    4497 scope.go:122] "RemoveContainer" containerID="16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0"
Oct 30 17:22:30.872532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.872513    4497 scope.go:122] "RemoveContainer" containerID="16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0"
Oct 30 17:22:30.872815 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:22:30.872797    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0\": container with ID starting with 16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0 not found: ID does not exist" containerID="16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0"
Oct 30 17:22:30.872931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.872826    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0"} err="failed to get container status \"16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0\": rpc error: code = NotFound desc = could not find container \"16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0\": container with ID starting with 16022fe07869914b7bbf06a66f99d7a9c04ceee1b98a6654043f34d02fecc8e0 not found: ID does not exist"
Oct 30 17:22:30.906344 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.906316    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"]
Oct 30 17:22:30.916558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:30.916534    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-abda73-6f7548f499-bngwf"]
Oct 30 17:22:32.300999 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:32.300966    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a124b7a1-5f18-4779-a839-3c7a8c37c5f8" path="/var/lib/kubelet/pods/a124b7a1-5f18-4779-a839-3c7a8c37c5f8/volumes"
Oct 30 17:22:51.916152 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:22:51.916117    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:22:55.702802 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:55.702758    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"]
Oct 30 17:22:55.703304 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:55.703017    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7" podUID="bd864eb4-5923-49b8-abd6-c62b8b5e4420" containerName="maas-api" containerID="cri-o://115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58" gracePeriod=30
Oct 30 17:22:56.003733 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.003641    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:22:56.033280 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.032349    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-maas-api-tls\") pod \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") "
Oct 30 17:22:56.033280 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.032415    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd864eb4-5923-49b8-abd6-c62b8b5e4420-kube-api-access-5qvkp\" (UniqueName: \"kubernetes.io/projected/bd864eb4-5923-49b8-abd6-c62b8b5e4420-kube-api-access-5qvkp\") pod \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") "
Oct 30 17:22:56.033280 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.032616    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd864eb4-5923-49b8-abd6-c62b8b5e4420-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd864eb4-5923-49b8-abd6-c62b8b5e4420-openshift-service-ca\") pod \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") "
Oct 30 17:22:56.033280 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.032758    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-metrics-certs\") pod \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\" (UID: \"bd864eb4-5923-49b8-abd6-c62b8b5e4420\") "
Oct 30 17:22:56.038223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.038162    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd864eb4-5923-49b8-abd6-c62b8b5e4420-openshift-service-ca" pod "bd864eb4-5923-49b8-abd6-c62b8b5e4420" (UID: "bd864eb4-5923-49b8-abd6-c62b8b5e4420"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:22:56.049164 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.049085    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd864eb4-5923-49b8-abd6-c62b8b5e4420-kube-api-access-5qvkp" pod "bd864eb4-5923-49b8-abd6-c62b8b5e4420" (UID: "bd864eb4-5923-49b8-abd6-c62b8b5e4420"). InnerVolumeSpecName "kube-api-access-5qvkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:22:56.081368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.073490    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-maas-api-tls" pod "bd864eb4-5923-49b8-abd6-c62b8b5e4420" (UID: "bd864eb4-5923-49b8-abd6-c62b8b5e4420"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:22:56.087530 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.087482    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-metrics-certs" pod "bd864eb4-5923-49b8-abd6-c62b8b5e4420" (UID: "bd864eb4-5923-49b8-abd6-c62b8b5e4420"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:22:56.099543 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.099505    4497 generic.go:356] "Generic (PLEG): container finished" podID="bd864eb4-5923-49b8-abd6-c62b8b5e4420" containerID="115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58" exitCode=0
Oct 30 17:22:56.099741 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.099701    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7" event={"ID":"bd864eb4-5923-49b8-abd6-c62b8b5e4420","Type":"ContainerDied","Data":"115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58"}
Oct 30 17:22:56.099819 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.099751    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7" event={"ID":"bd864eb4-5923-49b8-abd6-c62b8b5e4420","Type":"ContainerDied","Data":"89f6e3e0951627884116ac83a0a073277e0ab7a6a7570b06fbfe4867445da218"}
Oct 30 17:22:56.099819 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.099781    4497 scope.go:122] "RemoveContainer" containerID="115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58"
Oct 30 17:22:56.099959 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.099943    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"
Oct 30 17:22:56.117348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.117314    4497 scope.go:122] "RemoveContainer" containerID="115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58"
Oct 30 17:22:56.119874 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:22:56.118280    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58\": container with ID starting with 115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58 not found: ID does not exist" containerID="115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58"
Oct 30 17:22:56.119874 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.118322    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58"} err="failed to get container status \"115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58\": rpc error: code = NotFound desc = could not find container \"115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58\": container with ID starting with 115e0d7ff3c2363c0c23ad8bcc9d236462c91d95c03c533bd542ae66b2f28a58 not found: ID does not exist"
Oct 30 17:22:56.135497 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.135477    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:22:56.135497 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.135498    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd864eb4-5923-49b8-abd6-c62b8b5e4420-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:22:56.135629 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.135507    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qvkp\" (UniqueName: \"kubernetes.io/projected/bd864eb4-5923-49b8-abd6-c62b8b5e4420-kube-api-access-5qvkp\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:22:56.135629 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.135515    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd864eb4-5923-49b8-abd6-c62b8b5e4420-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:22:56.158646 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.158619    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"]
Oct 30 17:22:56.162669 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.162646    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-abda73-54fcfcd8f5-sz6l7"]
Oct 30 17:22:56.303441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:22:56.303405    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd864eb4-5923-49b8-abd6-c62b8b5e4420" path="/var/lib/kubelet/pods/bd864eb4-5923-49b8-abd6-c62b8b5e4420/volumes"
Oct 30 17:23:19.966932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:19.966877    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"]
Oct 30 17:23:19.967506 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:19.967117    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq" podUID="eae7a56f-53f0-42f1-8430-ab5572dbeb4c" containerName="maas-api" containerID="cri-o://b9b434f24a47092c6206e900b04fc001dcffc48fc3efd55eb9dbbef2f66551bb" gracePeriod=30
Oct 30 17:23:20.280706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.280488    4497 generic.go:356] "Generic (PLEG): container finished" podID="eae7a56f-53f0-42f1-8430-ab5572dbeb4c" containerID="b9b434f24a47092c6206e900b04fc001dcffc48fc3efd55eb9dbbef2f66551bb" exitCode=0
Oct 30 17:23:20.280706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.280549    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq" event={"ID":"eae7a56f-53f0-42f1-8430-ab5572dbeb4c","Type":"ContainerDied","Data":"b9b434f24a47092c6206e900b04fc001dcffc48fc3efd55eb9dbbef2f66551bb"}
Oct 30 17:23:20.280706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.280656    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq" event={"ID":"eae7a56f-53f0-42f1-8430-ab5572dbeb4c","Type":"ContainerDied","Data":"e7c799f5b649bfe5b023815aed7b72d568058e57e09956c9c62f738f891f3457"}
Oct 30 17:23:20.280706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.280681    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7c799f5b649bfe5b023815aed7b72d568058e57e09956c9c62f738f891f3457"
Oct 30 17:23:20.281646 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.281619    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 17:23:20.374616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.374585    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-maas-api-tls\") pod \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") "
Oct 30 17:23:20.374813 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.374722    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-openshift-service-ca\") pod \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") "
Oct 30 17:23:20.374894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.374825    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-kube-api-access-8nd25\" (UniqueName: \"kubernetes.io/projected/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-kube-api-access-8nd25\") pod \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") "
Oct 30 17:23:20.374894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.374877    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-metrics-certs\") pod \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\" (UID: \"eae7a56f-53f0-42f1-8430-ab5572dbeb4c\") "
Oct 30 17:23:20.375203 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.375158    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-openshift-service-ca" pod "eae7a56f-53f0-42f1-8430-ab5572dbeb4c" (UID: "eae7a56f-53f0-42f1-8430-ab5572dbeb4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:23:20.376077 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.376049    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:20.377447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.377420    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-metrics-certs" pod "eae7a56f-53f0-42f1-8430-ab5572dbeb4c" (UID: "eae7a56f-53f0-42f1-8430-ab5572dbeb4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:23:20.377662 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.377632    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-maas-api-tls" pod "eae7a56f-53f0-42f1-8430-ab5572dbeb4c" (UID: "eae7a56f-53f0-42f1-8430-ab5572dbeb4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:23:20.377937 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.377919    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-kube-api-access-8nd25" pod "eae7a56f-53f0-42f1-8430-ab5572dbeb4c" (UID: "eae7a56f-53f0-42f1-8430-ab5572dbeb4c"). InnerVolumeSpecName "kube-api-access-8nd25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:20.479627 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.479538    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:20.480050 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.480031    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nd25\" (UniqueName: \"kubernetes.io/projected/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-kube-api-access-8nd25\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:20.480177 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:20.480166    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eae7a56f-53f0-42f1-8430-ab5572dbeb4c-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:21.288745 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:21.288715    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"
Oct 30 17:23:21.333842 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:21.333813    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"]
Oct 30 17:23:21.337956 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:21.337931    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-dd715e-cb98bfcf-7s6nq"]
Oct 30 17:23:21.916625 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:23:21.916589    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:23:22.300907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:22.300875    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae7a56f-53f0-42f1-8430-ab5572dbeb4c" path="/var/lib/kubelet/pods/eae7a56f-53f0-42f1-8430-ab5572dbeb4c/volumes"
Oct 30 17:23:36.508559 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:36.508483    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"]
Oct 30 17:23:39.258546 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:39.258512    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"]
Oct 30 17:23:39.292141 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:39.292109    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"]
Oct 30 17:23:41.509639 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:41.509597    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p" podUID="bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" containerName="payload-pre-processing" containerID="cri-o://f846818075fee527557a75f4ed189c3769288be7fcc54b29d240b121d173a92a" gracePeriod=25
Oct 30 17:23:44.259707 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.259651    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg" podUID="e90987f2-2ff8-49cb-a3a9-d4e398ee1928" containerName="payload-processing" containerID="cri-o://87866cfcabbc8cfd783617bb3d0a6277595a73248596eae14bb09d25d63481a5" gracePeriod=25
Oct 30 17:23:44.292966 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.292933    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr" podUID="72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" containerName="payload-pre-processing" containerID="cri-o://c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093" gracePeriod=25
Oct 30 17:23:44.473543 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.473507    4497 generic.go:356] "Generic (PLEG): container finished" podID="e90987f2-2ff8-49cb-a3a9-d4e398ee1928" containerID="87866cfcabbc8cfd783617bb3d0a6277595a73248596eae14bb09d25d63481a5" exitCode=0
Oct 30 17:23:44.473709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.473577    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg" event={"ID":"e90987f2-2ff8-49cb-a3a9-d4e398ee1928","Type":"ContainerDied","Data":"87866cfcabbc8cfd783617bb3d0a6277595a73248596eae14bb09d25d63481a5"}
Oct 30 17:23:44.533458 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.533432    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 17:23:44.672675 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.672642    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-kube-api-access-5sq58\" (UniqueName: \"kubernetes.io/projected/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-kube-api-access-5sq58\") pod \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\" (UID: \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\") "
Oct 30 17:23:44.674222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.672863    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-plugins-config-volume\") pod \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\" (UID: \"e90987f2-2ff8-49cb-a3a9-d4e398ee1928\") "
Oct 30 17:23:44.674222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.674037    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-plugins-config-volume" pod "e90987f2-2ff8-49cb-a3a9-d4e398ee1928" (UID: "e90987f2-2ff8-49cb-a3a9-d4e398ee1928"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:23:44.676775 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.676667    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-kube-api-access-5sq58" pod "e90987f2-2ff8-49cb-a3a9-d4e398ee1928" (UID: "e90987f2-2ff8-49cb-a3a9-d4e398ee1928"). InnerVolumeSpecName "kube-api-access-5sq58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:44.775453 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.775377    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sq58\" (UniqueName: \"kubernetes.io/projected/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-kube-api-access-5sq58\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:44.775453 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:44.775406    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90987f2-2ff8-49cb-a3a9-d4e398ee1928-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:45.481256 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:45.481232    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"
Oct 30 17:23:45.481703 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:45.481223    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg" event={"ID":"e90987f2-2ff8-49cb-a3a9-d4e398ee1928","Type":"ContainerDied","Data":"305e81e34de0e1fd4cf8c998022f8136b7b7e98ad496e76a0d07086f11f5e8b1"}
Oct 30 17:23:45.481703 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:45.481384    4497 scope.go:122] "RemoveContainer" containerID="87866cfcabbc8cfd783617bb3d0a6277595a73248596eae14bb09d25d63481a5"
Oct 30 17:23:45.526792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:45.526767    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"]
Oct 30 17:23:45.535610 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:45.535591    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8bff6e-76fb58c874-tw2gg"]
Oct 30 17:23:46.301875 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:46.301839    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90987f2-2ff8-49cb-a3a9-d4e398ee1928" path="/var/lib/kubelet/pods/e90987f2-2ff8-49cb-a3a9-d4e398ee1928/volumes"
Oct 30 17:23:47.711967 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:47.711925    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p" podUID="bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:23:47.952415 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:47.952383    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"]
Oct 30 17:23:50.143909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:50.143864    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr" podUID="72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:23:50.720755 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:50.720722    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"]
Oct 30 17:23:50.720958 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:50.720931    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy" containerID="cri-o://db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14" gracePeriod=30
Oct 30 17:23:51.916988 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:23:51.916955    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:23:52.024726 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:52.024698    4497 scope.go:122] "RemoveContainer" containerID="b9b434f24a47092c6206e900b04fc001dcffc48fc3efd55eb9dbbef2f66551bb"
Oct 30 17:23:52.953619 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:52.953577    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz" podUID="b1915fda-de7a-4552-a52d-469a2deb8ac2" containerName="payload-pre-processing" containerID="cri-o://517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331" gracePeriod=25
Oct 30 17:23:53.761254 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:53.761212    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz" podUID="b1915fda-de7a-4552-a52d-469a2deb8ac2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:23:55.988704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:55.988679    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:23:56.028132 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028097    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-data\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.028304 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028202    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-envoy\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.028304 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028265    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-credential-socket\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.028425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028304    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-kube-api-access-6r5s5\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-kube-api-access-6r5s5\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.028425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028380    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-certs\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.028425 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028413    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3603e008-6a64-4524-9f75-f28a32a8f0f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3603e008-6a64-4524-9f75-f28a32a8f0f2-istiod-ca-cert\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.028567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028455    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-podinfo\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.028567 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028510    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-socket\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.028661 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.028597    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-token\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-token\") pod \"3603e008-6a64-4524-9f75-f28a32a8f0f2\" (UID: \"3603e008-6a64-4524-9f75-f28a32a8f0f2\") "
Oct 30 17:23:56.029631 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.029602    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-credential-socket" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:56.029821 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.029612    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-socket" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:56.029935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.029630    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-certs" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:56.029935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.029743    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-data" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:56.029935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.029803    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3603e008-6a64-4524-9f75-f28a32a8f0f2-istiod-ca-cert" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:23:56.030125 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.030039    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.030125 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.030063    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.030125 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.030078    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.030125 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.030091    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3603e008-6a64-4524-9f75-f28a32a8f0f2-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.030125 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.030104    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.031702 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.031677    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-token" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:56.031801 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.031698    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-podinfo" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:23:56.031955 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.031923    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-envoy" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:23:56.031955 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.031927    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-kube-api-access-6r5s5" pod "3603e008-6a64-4524-9f75-f28a32a8f0f2" (UID: "3603e008-6a64-4524-9f75-f28a32a8f0f2"). InnerVolumeSpecName "kube-api-access-6r5s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:23:56.131555 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.131523    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.131555 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.131553    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.131752 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.131564    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3603e008-6a64-4524-9f75-f28a32a8f0f2-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.131752 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.131576    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5s5\" (UniqueName: \"kubernetes.io/projected/3603e008-6a64-4524-9f75-f28a32a8f0f2-kube-api-access-6r5s5\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:23:56.560058 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.559978    4497 generic.go:356] "Generic (PLEG): container finished" podID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerID="db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14" exitCode=0
Oct 30 17:23:56.560058 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.560042    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"
Oct 30 17:23:56.560270 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.560058    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" event={"ID":"3603e008-6a64-4524-9f75-f28a32a8f0f2","Type":"ContainerDied","Data":"db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14"}
Oct 30 17:23:56.560270 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.560116    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm" event={"ID":"3603e008-6a64-4524-9f75-f28a32a8f0f2","Type":"ContainerDied","Data":"a14d2f53145280e6f32248fd41dea53c67796b9a8019a916ae12f0efc5972f38"}
Oct 30 17:23:56.560270 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.560138    4497 scope.go:122] "RemoveContainer" containerID="db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14"
Oct 30 17:23:56.571065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.571046    4497 scope.go:122] "RemoveContainer" containerID="db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14"
Oct 30 17:23:56.571376 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:23:56.571359    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14\": container with ID starting with db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14 not found: ID does not exist" containerID="db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14"
Oct 30 17:23:56.571429 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.571382    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14"} err="failed to get container status \"db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14\": rpc error: code = NotFound desc = could not find container \"db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14\": container with ID starting with db18d17ef4598e7d58b10d383c5f08d0d65109b9f20e0588294f0290ad2c7c14 not found: ID does not exist"
Oct 30 17:23:56.606252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.606230    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"]
Oct 30 17:23:56.615379 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:56.615360    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6q6mm"]
Oct 30 17:23:57.712069 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:57.712034    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p" podUID="bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:23:58.300599 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:23:58.300565    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" path="/var/lib/kubelet/pods/3603e008-6a64-4524-9f75-f28a32a8f0f2/volumes"
Oct 30 17:24:00.144271 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:00.144236    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr" podUID="72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:24:01.633872 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:01.633843    4497 generic.go:356] "Generic (PLEG): container finished" podID="bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" containerID="f846818075fee527557a75f4ed189c3769288be7fcc54b29d240b121d173a92a" exitCode=0
Oct 30 17:24:01.634214 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:01.633912    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p" event={"ID":"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b","Type":"ContainerDied","Data":"f846818075fee527557a75f4ed189c3769288be7fcc54b29d240b121d173a92a"}
Oct 30 17:24:01.817850 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:01.817822    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:24:01.908063 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:01.908030    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-kube-api-access-bnndq\" (UniqueName: \"kubernetes.io/projected/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-kube-api-access-bnndq\") pod \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\" (UID: \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\") "
Oct 30 17:24:01.908310 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:01.908294    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-plugins-config-volume\") pod \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\" (UID: \"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b\") "
Oct 30 17:24:01.908684 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:01.908658    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-plugins-config-volume" pod "bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" (UID: "bc660447-d7c5-4f66-8fd7-460e4b6f5f8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:01.910266 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:01.910246    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-kube-api-access-bnndq" pod "bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" (UID: "bc660447-d7c5-4f66-8fd7-460e4b6f5f8b"). InnerVolumeSpecName "kube-api-access-bnndq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:02.010348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:02.010306    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:02.010348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:02.010339    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnndq\" (UniqueName: \"kubernetes.io/projected/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b-kube-api-access-bnndq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:02.642243 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:02.642166    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"
Oct 30 17:24:02.642243 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:02.642229    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p" event={"ID":"bc660447-d7c5-4f66-8fd7-460e4b6f5f8b","Type":"ContainerDied","Data":"04b724f9f0d5fb2bdc1534a6abe1eec5dc2a2d8a304aa4073b0c81974a50aa79"}
Oct 30 17:24:02.642744 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:02.642288    4497 scope.go:122] "RemoveContainer" containerID="f846818075fee527557a75f4ed189c3769288be7fcc54b29d240b121d173a92a"
Oct 30 17:24:02.687584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:02.687559    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"]
Oct 30 17:24:02.696884 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:02.696853    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6e08bd-7cd7fc5575-ltb7p"]
Oct 30 17:24:03.761621 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:03.761585    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz" podUID="b1915fda-de7a-4552-a52d-469a2deb8ac2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:24:04.301030 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.301002    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" path="/var/lib/kubelet/pods/bc660447-d7c5-4f66-8fd7-460e4b6f5f8b/volumes"
Oct 30 17:24:04.579272 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.579244    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 17:24:04.644990 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.644954    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-kube-api-access-h565w\" (UniqueName: \"kubernetes.io/projected/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-kube-api-access-h565w\") pod \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\" (UID: \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\") "
Oct 30 17:24:04.645325 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.645300    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-plugins-config-volume\") pod \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\" (UID: \"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59\") "
Oct 30 17:24:04.646263 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.646237    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-plugins-config-volume" pod "72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" (UID: "72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:04.649198 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.649159    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-kube-api-access-h565w" pod "72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" (UID: "72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59"). InnerVolumeSpecName "kube-api-access-h565w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:04.658005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.657977    4497 generic.go:356] "Generic (PLEG): container finished" podID="72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" containerID="c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093" exitCode=0
Oct 30 17:24:04.658127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.658110    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr" event={"ID":"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59","Type":"ContainerDied","Data":"c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093"}
Oct 30 17:24:04.658201 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.658165    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr" event={"ID":"72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59","Type":"ContainerDied","Data":"7751f8bc0d16931319b6e70fea283b19b4282094c0f766363d896aa5722ef446"}
Oct 30 17:24:04.658253 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.658239    4497 scope.go:122] "RemoveContainer" containerID="c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093"
Oct 30 17:24:04.658370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.658356    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"
Oct 30 17:24:04.675256 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.675234    4497 scope.go:122] "RemoveContainer" containerID="c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093"
Oct 30 17:24:04.675505 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:04.675486    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093\": container with ID starting with c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093 not found: ID does not exist" containerID="c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093"
Oct 30 17:24:04.675601 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.675513    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093"} err="failed to get container status \"c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093\": rpc error: code = NotFound desc = could not find container \"c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093\": container with ID starting with c88df079850ef6baa05e110eee3f507d94248b3af4dd8971ab221d9232ae2093 not found: ID does not exist"
Oct 30 17:24:04.712489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.712463    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"]
Oct 30 17:24:04.716770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.716748    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8bff6e-59f949f595-ngmnr"]
Oct 30 17:24:04.747390 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.747368    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:04.747390 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:04.747388    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h565w\" (UniqueName: \"kubernetes.io/projected/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59-kube-api-access-h565w\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:06.300968 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:06.300936    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" path="/var/lib/kubelet/pods/72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59/volumes"
Oct 30 17:24:06.424855 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:06.424815    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"]
Oct 30 17:24:06.425129 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:06.425097    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy" containerID="cri-o://856ffacb7068817509bc3fad08aa22c91038ae5ae279f6aef8b9cd9d40bf42e3" gracePeriod=30
Oct 30 17:24:10.614762 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:10.614726    4497 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:24:10.615240 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:10.614773    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:24:11.705707 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.705677    4497 generic.go:356] "Generic (PLEG): container finished" podID="0191197b-1734-4678-9304-7bfff05b46ef" containerID="856ffacb7068817509bc3fad08aa22c91038ae5ae279f6aef8b9cd9d40bf42e3" exitCode=0
Oct 30 17:24:11.706089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.705750    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" event={"ID":"0191197b-1734-4678-9304-7bfff05b46ef","Type":"ContainerDied","Data":"856ffacb7068817509bc3fad08aa22c91038ae5ae279f6aef8b9cd9d40bf42e3"}
Oct 30 17:24:11.706089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.705802    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7" event={"ID":"0191197b-1734-4678-9304-7bfff05b46ef","Type":"ContainerDied","Data":"2a9730b5df3e3b4d1ce024642d37b8b25b5e7770c717689c3e268566c46aab5b"}
Oct 30 17:24:11.706089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.705818    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a9730b5df3e3b4d1ce024642d37b8b25b5e7770c717689c3e268566c46aab5b"
Oct 30 17:24:11.711158 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.711137    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:24:11.840390 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840355    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0191197b-1734-4678-9304-7bfff05b46ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0191197b-1734-4678-9304-7bfff05b46ef-istiod-ca-cert\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.840593 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840424    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-credential-socket\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.840593 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840484    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-kube-api-access-rb88k\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-kube-api-access-rb88k\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.840593 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840513    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-socket\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.840781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840654    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-certs\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.840781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840732    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0191197b-1734-4678-9304-7bfff05b46ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0191197b-1734-4678-9304-7bfff05b46ef-istio-podinfo\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.840781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840750    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-credential-socket" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:11.840946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840753    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0191197b-1734-4678-9304-7bfff05b46ef-istiod-ca-cert" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:11.840946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840818    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-envoy\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.840946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840883    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-socket" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:11.840946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840928    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-istio-token\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-istio-token\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.841172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840941    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-certs" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:11.841172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.840971    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-data\") pod \"0191197b-1734-4678-9304-7bfff05b46ef\" (UID: \"0191197b-1734-4678-9304-7bfff05b46ef\") "
Oct 30 17:24:11.841312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.841258    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-data" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:11.841586 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.841568    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:11.841586 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.841586    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0191197b-1734-4678-9304-7bfff05b46ef-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:11.841768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.841595    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:11.841768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.841604    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:11.841768 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.841612    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:11.842866 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.842831    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0191197b-1734-4678-9304-7bfff05b46ef-istio-podinfo" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:24:11.842983 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.842931    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-envoy" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:11.843314 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.843296    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-kube-api-access-rb88k" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "kube-api-access-rb88k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:11.843394 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.843322    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-istio-token" pod "0191197b-1734-4678-9304-7bfff05b46ef" (UID: "0191197b-1734-4678-9304-7bfff05b46ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:11.943391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.943324    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:11.943391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.943351    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb88k\" (UniqueName: \"kubernetes.io/projected/0191197b-1734-4678-9304-7bfff05b46ef-kube-api-access-rb88k\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:11.943391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.943362    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0191197b-1734-4678-9304-7bfff05b46ef-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:11.943391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:11.943371    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0191197b-1734-4678-9304-7bfff05b46ef-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:12.713121 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:12.713052    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"
Oct 30 17:24:12.767416 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:12.767386    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"]
Oct 30 17:24:12.776502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:12.776476    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-5lvs7"]
Oct 30 17:24:13.228779 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.228721    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:24:13.363989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.363960    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1915fda-de7a-4552-a52d-469a2deb8ac2-kube-api-access-k274c\" (UniqueName: \"kubernetes.io/projected/b1915fda-de7a-4552-a52d-469a2deb8ac2-kube-api-access-k274c\") pod \"b1915fda-de7a-4552-a52d-469a2deb8ac2\" (UID: \"b1915fda-de7a-4552-a52d-469a2deb8ac2\") "
Oct 30 17:24:13.364159 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.364061    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1915fda-de7a-4552-a52d-469a2deb8ac2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1915fda-de7a-4552-a52d-469a2deb8ac2-plugins-config-volume\") pod \"b1915fda-de7a-4552-a52d-469a2deb8ac2\" (UID: \"b1915fda-de7a-4552-a52d-469a2deb8ac2\") "
Oct 30 17:24:13.364439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.364417    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1915fda-de7a-4552-a52d-469a2deb8ac2-plugins-config-volume" pod "b1915fda-de7a-4552-a52d-469a2deb8ac2" (UID: "b1915fda-de7a-4552-a52d-469a2deb8ac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:13.366113 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.366093    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1915fda-de7a-4552-a52d-469a2deb8ac2-kube-api-access-k274c" pod "b1915fda-de7a-4552-a52d-469a2deb8ac2" (UID: "b1915fda-de7a-4552-a52d-469a2deb8ac2"). InnerVolumeSpecName "kube-api-access-k274c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:13.465723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.465696    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k274c\" (UniqueName: \"kubernetes.io/projected/b1915fda-de7a-4552-a52d-469a2deb8ac2-kube-api-access-k274c\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:13.465723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.465720    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1915fda-de7a-4552-a52d-469a2deb8ac2-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:13.721128 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.721087    4497 generic.go:356] "Generic (PLEG): container finished" podID="b1915fda-de7a-4552-a52d-469a2deb8ac2" containerID="517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331" exitCode=0
Oct 30 17:24:13.721128 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.721123    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"
Oct 30 17:24:13.721715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.721170    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz" event={"ID":"b1915fda-de7a-4552-a52d-469a2deb8ac2","Type":"ContainerDied","Data":"517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331"}
Oct 30 17:24:13.721715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.721259    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz" event={"ID":"b1915fda-de7a-4552-a52d-469a2deb8ac2","Type":"ContainerDied","Data":"5e5420ef6972da8addcd639eae6952014ec53274fee12a23064372985df930e5"}
Oct 30 17:24:13.721715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.721294    4497 scope.go:122] "RemoveContainer" containerID="517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331"
Oct 30 17:24:13.732255 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.732237    4497 scope.go:122] "RemoveContainer" containerID="517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331"
Oct 30 17:24:13.732518 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:13.732499    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331\": container with ID starting with 517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331 not found: ID does not exist" containerID="517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331"
Oct 30 17:24:13.732571 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.732526    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331"} err="failed to get container status \"517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331\": rpc error: code = NotFound desc = could not find container \"517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331\": container with ID starting with 517ab92e88d4facee3192cd10e1d8ff836d64af8f7648ba3bf05a33e5070a331 not found: ID does not exist"
Oct 30 17:24:13.765989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.765966    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"]
Oct 30 17:24:13.774901 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:13.774880    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9afbb2-8fb76564c-c6rdz"]
Oct 30 17:24:14.301511 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:14.301472    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0191197b-1734-4678-9304-7bfff05b46ef" path="/var/lib/kubelet/pods/0191197b-1734-4678-9304-7bfff05b46ef/volumes"
Oct 30 17:24:14.302000 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:14.301977    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1915fda-de7a-4552-a52d-469a2deb8ac2" path="/var/lib/kubelet/pods/b1915fda-de7a-4552-a52d-469a2deb8ac2/volumes"
Oct 30 17:24:17.549369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:17.549331    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"]
Oct 30 17:24:17.581166 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:17.581125    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"]
Oct 30 17:24:21.915811 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:21.915777    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:24:22.550643 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.550597    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47" podUID="066e972e-b5f4-4778-985e-e4a3174f4bfb" containerName="payload-processing" containerID="cri-o://fd7e81a5c229298c1d71da36fbc247cab8437a9e19e87202db296c678a871b46" gracePeriod=25
Oct 30 17:24:22.581927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.581880    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j" podUID="f7d643d6-6956-4378-aab5-b50f5f3cfc4a" containerName="payload-pre-processing" containerID="cri-o://7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af" gracePeriod=25
Oct 30 17:24:22.785941 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.785900    4497 generic.go:356] "Generic (PLEG): container finished" podID="066e972e-b5f4-4778-985e-e4a3174f4bfb" containerID="fd7e81a5c229298c1d71da36fbc247cab8437a9e19e87202db296c678a871b46" exitCode=0
Oct 30 17:24:22.786126 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.785941    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47" event={"ID":"066e972e-b5f4-4778-985e-e4a3174f4bfb","Type":"ContainerDied","Data":"fd7e81a5c229298c1d71da36fbc247cab8437a9e19e87202db296c678a871b46"}
Oct 30 17:24:22.816368 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.816317    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:24:22.889331 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.889294    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066e972e-b5f4-4778-985e-e4a3174f4bfb-kube-api-access-nrzx8\" (UniqueName: \"kubernetes.io/projected/066e972e-b5f4-4778-985e-e4a3174f4bfb-kube-api-access-nrzx8\") pod \"066e972e-b5f4-4778-985e-e4a3174f4bfb\" (UID: \"066e972e-b5f4-4778-985e-e4a3174f4bfb\") "
Oct 30 17:24:22.889547 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.889525    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066e972e-b5f4-4778-985e-e4a3174f4bfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066e972e-b5f4-4778-985e-e4a3174f4bfb-plugins-config-volume\") pod \"066e972e-b5f4-4778-985e-e4a3174f4bfb\" (UID: \"066e972e-b5f4-4778-985e-e4a3174f4bfb\") "
Oct 30 17:24:22.889890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.889867    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066e972e-b5f4-4778-985e-e4a3174f4bfb-plugins-config-volume" pod "066e972e-b5f4-4778-985e-e4a3174f4bfb" (UID: "066e972e-b5f4-4778-985e-e4a3174f4bfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:22.890127 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.890109    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/066e972e-b5f4-4778-985e-e4a3174f4bfb-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:22.891963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.891942    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066e972e-b5f4-4778-985e-e4a3174f4bfb-kube-api-access-nrzx8" pod "066e972e-b5f4-4778-985e-e4a3174f4bfb" (UID: "066e972e-b5f4-4778-985e-e4a3174f4bfb"). InnerVolumeSpecName "kube-api-access-nrzx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:22.991343 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:22.991308    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrzx8\" (UniqueName: \"kubernetes.io/projected/066e972e-b5f4-4778-985e-e4a3174f4bfb-kube-api-access-nrzx8\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:23.389396 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:23.389362    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j" podUID="f7d643d6-6956-4378-aab5-b50f5f3cfc4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:24:23.793469 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:23.793438    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47" event={"ID":"066e972e-b5f4-4778-985e-e4a3174f4bfb","Type":"ContainerDied","Data":"7a0fc1891b3d07bc445bc5ed4ad9d97e83f78045f398dda6794a27f561840aa1"}
Oct 30 17:24:23.793469 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:23.793462    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"
Oct 30 17:24:23.793702 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:23.793496    4497 scope.go:122] "RemoveContainer" containerID="fd7e81a5c229298c1d71da36fbc247cab8437a9e19e87202db296c678a871b46"
Oct 30 17:24:23.840212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:23.840170    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"]
Oct 30 17:24:23.849479 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:23.849457    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-85eb87-6d9bbf6db4-7fr47"]
Oct 30 17:24:24.300625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:24.300592    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066e972e-b5f4-4778-985e-e4a3174f4bfb" path="/var/lib/kubelet/pods/066e972e-b5f4-4778-985e-e4a3174f4bfb/volumes"
Oct 30 17:24:33.389305 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:33.389260    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j" podUID="f7d643d6-6956-4378-aab5-b50f5f3cfc4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:24:34.126383 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:34.126348    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"]
Oct 30 17:24:34.126619 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:34.126595    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy" containerID="cri-o://4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3" gracePeriod=30
Oct 30 17:24:35.121345 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:35.121311    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"]
Oct 30 17:24:35.121836 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:35.121551    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy" containerID="cri-o://405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4" gracePeriod=30
Oct 30 17:24:36.986437 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:36.986401    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"]
Oct 30 17:24:36.986877 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:36.986669    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy" containerID="cri-o://6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5" gracePeriod=30
Oct 30 17:24:37.898590 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:37.898557    4497 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:24:37.898769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:37.898616    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:24:38.973066 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:38.973034    4497 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:24:38.973466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:38.973087    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:24:39.403544 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.403516    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 17:24:39.460592 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.460552    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-token\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-token\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.460763 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.460625    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-envoy\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.460763 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.460663    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-kube-api-access-8vxhp\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-kube-api-access-8vxhp\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.460763 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.460691    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-certs\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.460927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.460763    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-socket\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.460927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.460861    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-data\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.460927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.460907    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-credential-socket\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.461091 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.461027    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff542801-6f7a-4588-9f8b-dac74cae73eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff542801-6f7a-4588-9f8b-dac74cae73eb-istiod-ca-cert\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.461091 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.461080    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-podinfo\") pod \"ff542801-6f7a-4588-9f8b-dac74cae73eb\" (UID: \"ff542801-6f7a-4588-9f8b-dac74cae73eb\") "
Oct 30 17:24:39.461231 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.461160    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-certs" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:39.461750 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.461716    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.462598 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.462410    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-data" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:39.462598 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.462575    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff542801-6f7a-4588-9f8b-dac74cae73eb-istiod-ca-cert" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:39.462986 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.462962    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-credential-socket" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:39.463107 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.463051    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-socket" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:39.464145 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.463464    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-token" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:39.464145 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.463607    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-kube-api-access-8vxhp" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "kube-api-access-8vxhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:39.464145 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.463839    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-podinfo" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:24:39.464963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.464943    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-envoy" pod "ff542801-6f7a-4588-9f8b-dac74cae73eb" (UID: "ff542801-6f7a-4588-9f8b-dac74cae73eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:39.504753 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.504726    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"]
Oct 30 17:24:39.514424 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.514399    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"]
Oct 30 17:24:39.563633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.563610    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.563633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.563631    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.563800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.563640    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxhp\" (UniqueName: \"kubernetes.io/projected/ff542801-6f7a-4588-9f8b-dac74cae73eb-kube-api-access-8vxhp\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.563800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.563649    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.563800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.563659    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.563800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.563669    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff542801-6f7a-4588-9f8b-dac74cae73eb-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.563800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.563678    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff542801-6f7a-4588-9f8b-dac74cae73eb-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.563800 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.563687    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff542801-6f7a-4588-9f8b-dac74cae73eb-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:39.930735 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.930659    4497 generic.go:356] "Generic (PLEG): container finished" podID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerID="4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3" exitCode=0
Oct 30 17:24:39.930735 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.930712    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"
Oct 30 17:24:39.930946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.930738    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" event={"ID":"ff542801-6f7a-4588-9f8b-dac74cae73eb","Type":"ContainerDied","Data":"4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3"}
Oct 30 17:24:39.930946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.930789    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx" event={"ID":"ff542801-6f7a-4588-9f8b-dac74cae73eb","Type":"ContainerDied","Data":"cf79d01a6b97a53e410ffa94cc1a5f3e02c17670710918aaf693d6a7ccc3ae4b"}
Oct 30 17:24:39.930946 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.930811    4497 scope.go:122] "RemoveContainer" containerID="4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3"
Oct 30 17:24:39.942170 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.942153    4497 scope.go:122] "RemoveContainer" containerID="5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7"
Oct 30 17:24:39.951588 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.951498    4497 scope.go:122] "RemoveContainer" containerID="4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3"
Oct 30 17:24:39.951777 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:39.951757    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3\": container with ID starting with 4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3 not found: ID does not exist" containerID="4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3"
Oct 30 17:24:39.951858 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.951788    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3"} err="failed to get container status \"4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3\": rpc error: code = NotFound desc = could not find container \"4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3\": container with ID starting with 4aef04c938dfe9a3434acbfc724fd5ee897dc1fdf5b98139a07d124b834389c3 not found: ID does not exist"
Oct 30 17:24:39.951858 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.951813    4497 scope.go:122] "RemoveContainer" containerID="5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7"
Oct 30 17:24:39.952071 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:39.952055    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7\": container with ID starting with 5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7 not found: ID does not exist" containerID="5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7"
Oct 30 17:24:39.952123 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.952079    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7"} err="failed to get container status \"5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7\": rpc error: code = NotFound desc = could not find container \"5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7\": container with ID starting with 5a2bf668a8b7cdf918a504a2d13808ec6f307da6fa6718601e6a91f5fa707fb7 not found: ID does not exist"
Oct 30 17:24:39.975601 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.975579    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"]
Oct 30 17:24:39.985337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:39.985314    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8bff6e-openshift-default-d68969585-kjhmx"]
Oct 30 17:24:40.302553 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.302253    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" path="/var/lib/kubelet/pods/ff542801-6f7a-4588-9f8b-dac74cae73eb/volumes"
Oct 30 17:24:40.402977 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.402945    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:24:40.475520 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475490    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-token\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-token\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.475713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475550    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-envoy\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.475713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475600    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-data\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.475713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475619    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-kube-api-access-f459f\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-kube-api-access-f459f\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.475713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475695    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-certs\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.475943 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475734    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-credential-socket\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.475943 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475827    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-podinfo\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.475943 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475881    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istiod-ca-cert\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.475943 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475913    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-socket\") pod \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\" (UID: \"f3ee375e-34e5-45d1-b01c-61ce79ef4c33\") "
Oct 30 17:24:40.476152 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475957    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-data" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:40.476152 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.475992    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-certs" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:40.476333 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.476311    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-credential-socket" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:40.476402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.476313    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istiod-ca-cert" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:40.476547 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.476523    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-socket" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:40.476715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.476696    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.476792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.476722    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.476792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.476736    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.476792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.476749    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.476792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.476763    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.478223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.478167    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-kube-api-access-f459f" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "kube-api-access-f459f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:40.478336 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.478277    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-envoy" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:40.478396 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.478374    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-token" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:40.478899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.478875    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-podinfo" pod "f3ee375e-34e5-45d1-b01c-61ce79ef4c33" (UID: "f3ee375e-34e5-45d1-b01c-61ce79ef4c33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:24:40.578097 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.578065    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.578097 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.578093    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.578097 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.578102    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f459f\" (UniqueName: \"kubernetes.io/projected/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-kube-api-access-f459f\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.578342 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.578113    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ee375e-34e5-45d1-b01c-61ce79ef4c33-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:40.939032 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.938950    4497 generic.go:356] "Generic (PLEG): container finished" podID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerID="405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4" exitCode=0
Oct 30 17:24:40.939032 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.938992    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"
Oct 30 17:24:40.939287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.939031    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" event={"ID":"f3ee375e-34e5-45d1-b01c-61ce79ef4c33","Type":"ContainerDied","Data":"405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4"}
Oct 30 17:24:40.939287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.939108    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt" event={"ID":"f3ee375e-34e5-45d1-b01c-61ce79ef4c33","Type":"ContainerDied","Data":"4bace96c76d3164b5216350892c8ba3d2bd6a52314689469e32c7949d2f18b74"}
Oct 30 17:24:40.939287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.939132    4497 scope.go:122] "RemoveContainer" containerID="405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4"
Oct 30 17:24:40.950443 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.950427    4497 scope.go:122] "RemoveContainer" containerID="b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677"
Oct 30 17:24:40.959521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.959501    4497 scope.go:122] "RemoveContainer" containerID="405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4"
Oct 30 17:24:40.959756 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:40.959740    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4\": container with ID starting with 405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4 not found: ID does not exist" containerID="405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4"
Oct 30 17:24:40.959808 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.959764    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4"} err="failed to get container status \"405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4\": rpc error: code = NotFound desc = could not find container \"405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4\": container with ID starting with 405e00b120234356e0fc71f75f4de04325400782abfbb9195a65d690cd1badb4 not found: ID does not exist"
Oct 30 17:24:40.959808 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.959780    4497 scope.go:122] "RemoveContainer" containerID="b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677"
Oct 30 17:24:40.959981 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:40.959962    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677\": container with ID starting with b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677 not found: ID does not exist" containerID="b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677"
Oct 30 17:24:40.960023 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.959987    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677"} err="failed to get container status \"b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677\": rpc error: code = NotFound desc = could not find container \"b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677\": container with ID starting with b218ce3bf29a089d8bdbf9d4e9901bfc765c19a63740d82e606e52e6eaaa9677 not found: ID does not exist"
Oct 30 17:24:40.984380 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.984357    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"]
Oct 30 17:24:40.993545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:40.993523    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6e08bd-openshift-default-75d85f9c85-87qkt"]
Oct 30 17:24:42.249799 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.249745    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:24:42.302481 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.302446    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" path="/var/lib/kubelet/pods/f3ee375e-34e5-45d1-b01c-61ce79ef4c33/volumes"
Oct 30 17:24:42.400902 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.400877    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-kube-api-access-pq2zj\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-kube-api-access-pq2zj\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.400956    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-certs\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.400997    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-podinfo\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401023    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-token\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-token\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401089 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401069    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-data\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401347 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401275    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-credential-socket\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401408 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401367    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b635ded-7d16-47e4-a0c6-a71d0a131533-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b635ded-7d16-47e4-a0c6-a71d0a131533-istiod-ca-cert\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401425    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-socket\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401462 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401453    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-envoy\") pod \"6b635ded-7d16-47e4-a0c6-a71d0a131533\" (UID: \"6b635ded-7d16-47e4-a0c6-a71d0a131533\") "
Oct 30 17:24:42.401581 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401531    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-data" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:42.401672 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401648    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-certs" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:42.401742 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401720    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-socket" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:42.402306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.401878    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-credential-socket" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:42.402306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.402045    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b635ded-7d16-47e4-a0c6-a71d0a131533-istiod-ca-cert" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:42.402306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.402065    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.402306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.402088    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.402306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.402102    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.402306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.402115    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.404098 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.404073    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-envoy" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:42.404212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.404123    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-kube-api-access-pq2zj" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "kube-api-access-pq2zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:42.404385 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.404365    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-podinfo" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:24:42.404778 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.404738    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-token" pod "6b635ded-7d16-47e4-a0c6-a71d0a131533" (UID: "6b635ded-7d16-47e4-a0c6-a71d0a131533"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:42.503457 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.503409    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b635ded-7d16-47e4-a0c6-a71d0a131533-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.503457 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.503429    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.503457 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.503438    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2zj\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-kube-api-access-pq2zj\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.503457 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.503448    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.503457 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.503456    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b635ded-7d16-47e4-a0c6-a71d0a131533-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.831793 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.831767    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:24:42.909639 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.909603    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-kube-api-access-q9pjl\" (UniqueName: \"kubernetes.io/projected/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-kube-api-access-q9pjl\") pod \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\" (UID: \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\") "
Oct 30 17:24:42.909979 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.909955    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-plugins-config-volume\") pod \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\" (UID: \"f7d643d6-6956-4378-aab5-b50f5f3cfc4a\") "
Oct 30 17:24:42.910436 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.910411    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-plugins-config-volume" pod "f7d643d6-6956-4378-aab5-b50f5f3cfc4a" (UID: "f7d643d6-6956-4378-aab5-b50f5f3cfc4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:42.910535 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.910515    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:42.911747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.911724    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-kube-api-access-q9pjl" pod "f7d643d6-6956-4378-aab5-b50f5f3cfc4a" (UID: "f7d643d6-6956-4378-aab5-b50f5f3cfc4a"). InnerVolumeSpecName "kube-api-access-q9pjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:42.956060 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.956029    4497 generic.go:356] "Generic (PLEG): container finished" podID="f7d643d6-6956-4378-aab5-b50f5f3cfc4a" containerID="7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af" exitCode=0
Oct 30 17:24:42.956181 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.956070    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"
Oct 30 17:24:42.956181 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.956106    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j" event={"ID":"f7d643d6-6956-4378-aab5-b50f5f3cfc4a","Type":"ContainerDied","Data":"7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af"}
Oct 30 17:24:42.956181 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.956159    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j" event={"ID":"f7d643d6-6956-4378-aab5-b50f5f3cfc4a","Type":"ContainerDied","Data":"6faf111ee8e28a1a98179684f67f90906ec08abaf92d409f0a71642c21469d82"}
Oct 30 17:24:42.956341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.956203    4497 scope.go:122] "RemoveContainer" containerID="7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af"
Oct 30 17:24:42.957702 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.957674    4497 generic.go:356] "Generic (PLEG): container finished" podID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerID="6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5" exitCode=0
Oct 30 17:24:42.957818 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.957720    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"
Oct 30 17:24:42.957886 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.957814    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" event={"ID":"6b635ded-7d16-47e4-a0c6-a71d0a131533","Type":"ContainerDied","Data":"6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5"}
Oct 30 17:24:42.957886 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.957857    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc" event={"ID":"6b635ded-7d16-47e4-a0c6-a71d0a131533","Type":"ContainerDied","Data":"204ae1af1873797fd652e348388e006a322e97b2a404167e5e6bace113f5cd6c"}
Oct 30 17:24:42.967699 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.967685    4497 scope.go:122] "RemoveContainer" containerID="7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af"
Oct 30 17:24:42.968032 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:42.968007    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af\": container with ID starting with 7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af not found: ID does not exist" containerID="7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af"
Oct 30 17:24:42.968107 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.968039    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af"} err="failed to get container status \"7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af\": rpc error: code = NotFound desc = could not find container \"7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af\": container with ID starting with 7a47e8310854a03d78d927043b5a53f3ba1881023d7524677e97cd5d509971af not found: ID does not exist"
Oct 30 17:24:42.968107 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.968055    4497 scope.go:122] "RemoveContainer" containerID="6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5"
Oct 30 17:24:42.977086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.977072    4497 scope.go:122] "RemoveContainer" containerID="6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5"
Oct 30 17:24:42.977333 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:42.977315    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5\": container with ID starting with 6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5 not found: ID does not exist" containerID="6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5"
Oct 30 17:24:42.977380 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:42.977340    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5"} err="failed to get container status \"6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5\": rpc error: code = NotFound desc = could not find container \"6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5\": container with ID starting with 6cbe27b899b90e60a1c241e3ea412451ebb236beeb21b3243a288f127d60aeb5 not found: ID does not exist"
Oct 30 17:24:43.001519 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:43.001494    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"]
Oct 30 17:24:43.005697 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:43.005676    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-85eb87-66b6b5bb4c-5lm2j"]
Oct 30 17:24:43.012131 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:43.012107    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9pjl\" (UniqueName: \"kubernetes.io/projected/f7d643d6-6956-4378-aab5-b50f5f3cfc4a-kube-api-access-q9pjl\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:43.034792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:43.034771    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"]
Oct 30 17:24:43.038615 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:43.038597    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-msbxc"]
Oct 30 17:24:44.300561 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:44.300530    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" path="/var/lib/kubelet/pods/6b635ded-7d16-47e4-a0c6-a71d0a131533/volumes"
Oct 30 17:24:44.300959 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:44.300945    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d643d6-6956-4378-aab5-b50f5f3cfc4a" path="/var/lib/kubelet/pods/f7d643d6-6956-4378-aab5-b50f5f3cfc4a/volumes"
Oct 30 17:24:46.027520 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:46.027483    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"]
Oct 30 17:24:51.028658 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:51.028620    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl" podUID="525852cc-f693-4705-b159-d66720176b20" containerName="payload-processing" containerID="cri-o://47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62" gracePeriod=25
Oct 30 17:24:51.295439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:51.295386    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:24:51.335654 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:51.335628    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525852cc-f693-4705-b159-d66720176b20-kube-api-access-qm55p\" (UniqueName: \"kubernetes.io/projected/525852cc-f693-4705-b159-d66720176b20-kube-api-access-qm55p\") pod \"525852cc-f693-4705-b159-d66720176b20\" (UID: \"525852cc-f693-4705-b159-d66720176b20\") "
Oct 30 17:24:51.335872 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:51.335710    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525852cc-f693-4705-b159-d66720176b20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525852cc-f693-4705-b159-d66720176b20-plugins-config-volume\") pod \"525852cc-f693-4705-b159-d66720176b20\" (UID: \"525852cc-f693-4705-b159-d66720176b20\") "
Oct 30 17:24:51.336211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:51.336159    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525852cc-f693-4705-b159-d66720176b20-plugins-config-volume" pod "525852cc-f693-4705-b159-d66720176b20" (UID: "525852cc-f693-4705-b159-d66720176b20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:24:51.336415 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:51.336394    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525852cc-f693-4705-b159-d66720176b20-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:51.337993 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:51.337968    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525852cc-f693-4705-b159-d66720176b20-kube-api-access-qm55p" pod "525852cc-f693-4705-b159-d66720176b20" (UID: "525852cc-f693-4705-b159-d66720176b20"). InnerVolumeSpecName "kube-api-access-qm55p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:51.437441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:51.437413    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm55p\" (UniqueName: \"kubernetes.io/projected/525852cc-f693-4705-b159-d66720176b20-kube-api-access-qm55p\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:51.915660 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:51.915625    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:24:52.030957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.030922    4497 generic.go:356] "Generic (PLEG): container finished" podID="525852cc-f693-4705-b159-d66720176b20" containerID="47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62" exitCode=0
Oct 30 17:24:52.031931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.031665    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl" event={"ID":"525852cc-f693-4705-b159-d66720176b20","Type":"ContainerDied","Data":"47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"}
Oct 30 17:24:52.031931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.031731    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl" event={"ID":"525852cc-f693-4705-b159-d66720176b20","Type":"ContainerDied","Data":"bdf908c0773a00b3fbdc531800b904e11c9f6049fd1405b89dff6e9f3bb20fd0"}
Oct 30 17:24:52.031931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.031761    4497 scope.go:122] "RemoveContainer" containerID="47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"
Oct 30 17:24:52.032397 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.032253    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"
Oct 30 17:24:52.064086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.064062    4497 scope.go:122] "RemoveContainer" containerID="47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"
Oct 30 17:24:52.074130 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:52.074089    4497 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl_openshift-ingress_525852cc-f693-4705-b159-d66720176b20_0 in pod sandbox bdf908c0773a00b3fbdc531800b904e11c9f6049fd1405b89dff6e9f3bb20fd0 from index: no such id: '47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62'" containerID="47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"
Oct 30 17:24:52.074237 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:52.074139    4497 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl_openshift-ingress_525852cc-f693-4705-b159-d66720176b20_0 in pod sandbox bdf908c0773a00b3fbdc531800b904e11c9f6049fd1405b89dff6e9f3bb20fd0 from index: no such id: '47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62'" containerID="47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"
Oct 30 17:24:52.074337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.074320    4497 scope.go:122] "RemoveContainer" containerID="47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"
Oct 30 17:24:52.078680 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:52.078617    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62\": container with ID starting with 47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62 not found: ID does not exist" containerID="47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"
Oct 30 17:24:52.078680 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.078652    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62"} err="failed to get container status \"47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62\": rpc error: code = NotFound desc = could not find container \"47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62\": container with ID starting with 47bdfbadfb80249d63619bd3a7769d40eb224c3fc670361daf5df08aea6e3b62 not found: ID does not exist"
Oct 30 17:24:52.094773 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.094743    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"]
Oct 30 17:24:52.104051 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.104032    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bbc36f-556b5cbb66-ppngl"]
Oct 30 17:24:52.304204 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:52.304159    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525852cc-f693-4705-b159-d66720176b20" path="/var/lib/kubelet/pods/525852cc-f693-4705-b159-d66720176b20/volumes"
Oct 30 17:24:54.525909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.525853    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main" containerID="cri-o://70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432" gracePeriod=45
Oct 30 17:24:54.534337 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.534282    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main" containerID="cri-o://f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684" gracePeriod=45
Oct 30 17:24:54.840356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.840333    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 17:24:54.843530 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.843507    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 17:24:54.892504 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.892251    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a890f36-432f-4f13-b9c4-067c99f91990-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a890f36-432f-4f13-b9c4-067c99f91990-tls-certs\") pod \"1a890f36-432f-4f13-b9c4-067c99f91990\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") "
Oct 30 17:24:54.892504 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.892295    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-tmp-dir\") pod \"1a890f36-432f-4f13-b9c4-067c99f91990\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") "
Oct 30 17:24:54.892504 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.892329    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-model-cache\") pod \"f917b48c-109a-4c9b-8a32-be33e53b320a\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") "
Oct 30 17:24:54.892504 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.892396    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a890f36-432f-4f13-b9c4-067c99f91990-kube-api-access-gxwcs\" (UniqueName: \"kubernetes.io/projected/1a890f36-432f-4f13-b9c4-067c99f91990-kube-api-access-gxwcs\") pod \"1a890f36-432f-4f13-b9c4-067c99f91990\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") "
Oct 30 17:24:54.892504 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.892438    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f917b48c-109a-4c9b-8a32-be33e53b320a-kube-api-access-zg4gb\" (UniqueName: \"kubernetes.io/projected/f917b48c-109a-4c9b-8a32-be33e53b320a-kube-api-access-zg4gb\") pod \"f917b48c-109a-4c9b-8a32-be33e53b320a\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") "
Oct 30 17:24:54.892912 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.892867    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-model-cache" pod "f917b48c-109a-4c9b-8a32-be33e53b320a" (UID: "f917b48c-109a-4c9b-8a32-be33e53b320a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:54.893301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.892473    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-home\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-home\") pod \"1a890f36-432f-4f13-b9c4-067c99f91990\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") "
Oct 30 17:24:54.893301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.893065    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-tmp-dir" pod "1a890f36-432f-4f13-b9c4-067c99f91990" (UID: "1a890f36-432f-4f13-b9c4-067c99f91990"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:54.893301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.893119    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-dshm\") pod \"1a890f36-432f-4f13-b9c4-067c99f91990\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") "
Oct 30 17:24:54.893301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.893157    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f917b48c-109a-4c9b-8a32-be33e53b320a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f917b48c-109a-4c9b-8a32-be33e53b320a-tls-certs\") pod \"f917b48c-109a-4c9b-8a32-be33e53b320a\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") "
Oct 30 17:24:54.893301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.893240    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-tmp-dir\") pod \"f917b48c-109a-4c9b-8a32-be33e53b320a\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") "
Oct 30 17:24:54.893643 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.893361    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-dshm\") pod \"f917b48c-109a-4c9b-8a32-be33e53b320a\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") "
Oct 30 17:24:54.893643 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.893434    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-home\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-home\") pod \"f917b48c-109a-4c9b-8a32-be33e53b320a\" (UID: \"f917b48c-109a-4c9b-8a32-be33e53b320a\") "
Oct 30 17:24:54.893643 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.893459    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-model-cache\") pod \"1a890f36-432f-4f13-b9c4-067c99f91990\" (UID: \"1a890f36-432f-4f13-b9c4-067c99f91990\") "
Oct 30 17:24:54.894076 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.893892    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-home" pod "1a890f36-432f-4f13-b9c4-067c99f91990" (UID: "1a890f36-432f-4f13-b9c4-067c99f91990"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:54.894168 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.894111    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-model-cache" pod "1a890f36-432f-4f13-b9c4-067c99f91990" (UID: "1a890f36-432f-4f13-b9c4-067c99f91990"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:54.894376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.894286    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-home" pod "f917b48c-109a-4c9b-8a32-be33e53b320a" (UID: "f917b48c-109a-4c9b-8a32-be33e53b320a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:54.894376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.894323    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.894376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.894344    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.894376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.894356    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.894598 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.894445    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-tmp-dir" pod "f917b48c-109a-4c9b-8a32-be33e53b320a" (UID: "f917b48c-109a-4c9b-8a32-be33e53b320a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:54.909856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.909693    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f917b48c-109a-4c9b-8a32-be33e53b320a-tls-certs" pod "f917b48c-109a-4c9b-8a32-be33e53b320a" (UID: "f917b48c-109a-4c9b-8a32-be33e53b320a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:24:54.909856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.909728    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f917b48c-109a-4c9b-8a32-be33e53b320a-kube-api-access-zg4gb" pod "f917b48c-109a-4c9b-8a32-be33e53b320a" (UID: "f917b48c-109a-4c9b-8a32-be33e53b320a"). InnerVolumeSpecName "kube-api-access-zg4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:54.909856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.909829    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a890f36-432f-4f13-b9c4-067c99f91990-tls-certs" pod "1a890f36-432f-4f13-b9c4-067c99f91990" (UID: "1a890f36-432f-4f13-b9c4-067c99f91990"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:24:54.910268 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.910230    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-dshm" pod "1a890f36-432f-4f13-b9c4-067c99f91990" (UID: "1a890f36-432f-4f13-b9c4-067c99f91990"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:54.911228 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.911204    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-dshm" pod "f917b48c-109a-4c9b-8a32-be33e53b320a" (UID: "f917b48c-109a-4c9b-8a32-be33e53b320a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:24:54.911329 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.911310    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a890f36-432f-4f13-b9c4-067c99f91990-kube-api-access-gxwcs" pod "1a890f36-432f-4f13-b9c4-067c99f91990" (UID: "1a890f36-432f-4f13-b9c4-067c99f91990"). InnerVolumeSpecName "kube-api-access-gxwcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:24:54.995356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995324    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.995356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995350    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f917b48c-109a-4c9b-8a32-be33e53b320a-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.995356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995360    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.995584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995367    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.995584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995375    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f917b48c-109a-4c9b-8a32-be33e53b320a-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.995584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995382    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a890f36-432f-4f13-b9c4-067c99f91990-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.995584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995389    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a890f36-432f-4f13-b9c4-067c99f91990-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.995584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995397    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxwcs\" (UniqueName: \"kubernetes.io/projected/1a890f36-432f-4f13-b9c4-067c99f91990-kube-api-access-gxwcs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:54.995584 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:54.995406    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg4gb\" (UniqueName: \"kubernetes.io/projected/f917b48c-109a-4c9b-8a32-be33e53b320a-kube-api-access-zg4gb\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:24:55.052210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.052104    4497 generic.go:356] "Generic (PLEG): container finished" podID="1a890f36-432f-4f13-b9c4-067c99f91990" containerID="70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432" exitCode=0
Oct 30 17:24:55.052210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.052146    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"
Oct 30 17:24:55.052454 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.052198    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" event={"ID":"1a890f36-432f-4f13-b9c4-067c99f91990","Type":"ContainerDied","Data":"70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432"}
Oct 30 17:24:55.052454 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.052257    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j" event={"ID":"1a890f36-432f-4f13-b9c4-067c99f91990","Type":"ContainerDied","Data":"e3b199d8cd666ea1e3d1a8f058b33bf1677ef6259516862b6544eff76265292f"}
Oct 30 17:24:55.052454 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.052281    4497 scope.go:122] "RemoveContainer" containerID="70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432"
Oct 30 17:24:55.053545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.053525    4497 generic.go:356] "Generic (PLEG): container finished" podID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerID="f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684" exitCode=0
Oct 30 17:24:55.053680 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.053616    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"
Oct 30 17:24:55.053680 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.053659    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" event={"ID":"f917b48c-109a-4c9b-8a32-be33e53b320a","Type":"ContainerDied","Data":"f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684"}
Oct 30 17:24:55.053844 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.053692    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst" event={"ID":"f917b48c-109a-4c9b-8a32-be33e53b320a","Type":"ContainerDied","Data":"2d481f991108966ec8a772f396da6a3f46145b57df93cb5098cc85375ad76019"}
Oct 30 17:24:55.062989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.062969    4497 scope.go:122] "RemoveContainer" containerID="70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432"
Oct 30 17:24:55.063286 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:55.063265    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432\": container with ID starting with 70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432 not found: ID does not exist" containerID="70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432"
Oct 30 17:24:55.063356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.063294    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432"} err="failed to get container status \"70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432\": rpc error: code = NotFound desc = could not find container \"70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432\": container with ID starting with 70f999756c9e083693fbfc29daddd7ba0d40134a033708709e639170d3e6f432 not found: ID does not exist"
Oct 30 17:24:55.063356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.063311    4497 scope.go:122] "RemoveContainer" containerID="f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684"
Oct 30 17:24:55.074285 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.074265    4497 scope.go:122] "RemoveContainer" containerID="f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684"
Oct 30 17:24:55.074583 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:24:55.074559    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684\": container with ID starting with f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684 not found: ID does not exist" containerID="f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684"
Oct 30 17:24:55.074670 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.074591    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684"} err="failed to get container status \"f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684\": rpc error: code = NotFound desc = could not find container \"f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684\": container with ID starting with f50287ec9add35fdc735e8ba659f8cee8ac86b8b49891e3ca76e59e267009684 not found: ID does not exist"
Oct 30 17:24:55.096353 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.096325    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"]
Oct 30 17:24:55.101007 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.100984    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-w3-2d365c-kserve-57f8f765-2fd2j"]
Oct 30 17:24:55.141107 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.141079    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"]
Oct 30 17:24:55.144921 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:55.144896    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/premium-w3-2d365c-kserve-d98ffccc8-7lzst"]
Oct 30 17:24:56.300468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:56.300439    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" path="/var/lib/kubelet/pods/1a890f36-432f-4f13-b9c4-067c99f91990/volumes"
Oct 30 17:24:56.300907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:24:56.300888    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" path="/var/lib/kubelet/pods/f917b48c-109a-4c9b-8a32-be33e53b320a/volumes"
Oct 30 17:25:16.027855 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.027758    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"]
Oct 30 17:25:16.028451 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.028375    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7" podUID="ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" containerName="maas-api" containerID="cri-o://cd5fc009c510bba96c716b270b746f2b51b4bdeb04a65a0145b34d9a98d51c56" gracePeriod=30
Oct 30 17:25:16.186565 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.186520    4497 generic.go:356] "Generic (PLEG): container finished" podID="ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" containerID="cd5fc009c510bba96c716b270b746f2b51b4bdeb04a65a0145b34d9a98d51c56" exitCode=0
Oct 30 17:25:16.186723 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.186588    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7" event={"ID":"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e","Type":"ContainerDied","Data":"cd5fc009c510bba96c716b270b746f2b51b4bdeb04a65a0145b34d9a98d51c56"}
Oct 30 17:25:16.281635 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.281527    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:25:16.357819 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.356397    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-maas-api-tls\") pod \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") "
Oct 30 17:25:16.357819 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.356456    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-metrics-certs\") pod \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") "
Oct 30 17:25:16.357819 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.356514    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-kube-api-access-z624l\" (UniqueName: \"kubernetes.io/projected/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-kube-api-access-z624l\") pod \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") "
Oct 30 17:25:16.357819 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.356544    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-openshift-service-ca\") pod \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\" (UID: \"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e\") "
Oct 30 17:25:16.362578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.362552    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-openshift-service-ca" pod "ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" (UID: "ec157bad-0aa9-4abe-96fc-f1e7b7a4186e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:25:16.375670 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.375627    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-maas-api-tls" pod "ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" (UID: "ec157bad-0aa9-4abe-96fc-f1e7b7a4186e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:16.380211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.380161    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-kube-api-access-z624l" pod "ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" (UID: "ec157bad-0aa9-4abe-96fc-f1e7b7a4186e"). InnerVolumeSpecName "kube-api-access-z624l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:16.380342 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.380220    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-metrics-certs" pod "ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" (UID: "ec157bad-0aa9-4abe-96fc-f1e7b7a4186e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:16.458835 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.458795    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:16.458835 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.458837    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:16.459036 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.458853    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z624l\" (UniqueName: \"kubernetes.io/projected/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-kube-api-access-z624l\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:16.459036 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.458869    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:16.515839 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.515798    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"]
Oct 30 17:25:16.516072 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.516043    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5" podUID="b8aaa1be-6f0b-451f-9486-f631143c55ac" containerName="maas-api" containerID="cri-o://1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6" gracePeriod=30
Oct 30 17:25:16.758950 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.758906    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:25:16.864553 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.864524    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8aaa1be-6f0b-451f-9486-f631143c55ac-kube-api-access-s2dr6\" (UniqueName: \"kubernetes.io/projected/b8aaa1be-6f0b-451f-9486-f631143c55ac-kube-api-access-s2dr6\") pod \"b8aaa1be-6f0b-451f-9486-f631143c55ac\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") "
Oct 30 17:25:16.864721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.864595    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-metrics-certs\") pod \"b8aaa1be-6f0b-451f-9486-f631143c55ac\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") "
Oct 30 17:25:16.864721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.864629    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8aaa1be-6f0b-451f-9486-f631143c55ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaa1be-6f0b-451f-9486-f631143c55ac-openshift-service-ca\") pod \"b8aaa1be-6f0b-451f-9486-f631143c55ac\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") "
Oct 30 17:25:16.864824 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.864813    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-maas-api-tls\") pod \"b8aaa1be-6f0b-451f-9486-f631143c55ac\" (UID: \"b8aaa1be-6f0b-451f-9486-f631143c55ac\") "
Oct 30 17:25:16.865094 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.865062    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8aaa1be-6f0b-451f-9486-f631143c55ac-openshift-service-ca" pod "b8aaa1be-6f0b-451f-9486-f631143c55ac" (UID: "b8aaa1be-6f0b-451f-9486-f631143c55ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:25:16.865246 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.865229    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8aaa1be-6f0b-451f-9486-f631143c55ac-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:16.866856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.866830    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8aaa1be-6f0b-451f-9486-f631143c55ac-kube-api-access-s2dr6" pod "b8aaa1be-6f0b-451f-9486-f631143c55ac" (UID: "b8aaa1be-6f0b-451f-9486-f631143c55ac"). InnerVolumeSpecName "kube-api-access-s2dr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:16.866968 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.866855    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-maas-api-tls" pod "b8aaa1be-6f0b-451f-9486-f631143c55ac" (UID: "b8aaa1be-6f0b-451f-9486-f631143c55ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:16.866968 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.866873    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-metrics-certs" pod "b8aaa1be-6f0b-451f-9486-f631143c55ac" (UID: "b8aaa1be-6f0b-451f-9486-f631143c55ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:16.927126 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.927071    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"]
Oct 30 17:25:16.978211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.971323    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:16.978211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.971357    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2dr6\" (UniqueName: \"kubernetes.io/projected/b8aaa1be-6f0b-451f-9486-f631143c55ac-kube-api-access-s2dr6\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:16.978211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:16.971373    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8aaa1be-6f0b-451f-9486-f631143c55ac-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:17.205572 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.205478    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7" event={"ID":"ec157bad-0aa9-4abe-96fc-f1e7b7a4186e","Type":"ContainerDied","Data":"075238c75612f048ce9bcfbf49619bfbed1b1c6a4da46c8e728bea2b441db464"}
Oct 30 17:25:17.206018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.205600    4497 scope.go:122] "RemoveContainer" containerID="cd5fc009c510bba96c716b270b746f2b51b4bdeb04a65a0145b34d9a98d51c56"
Oct 30 17:25:17.206018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.205879    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"
Oct 30 17:25:17.217044 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.216854    4497 generic.go:356] "Generic (PLEG): container finished" podID="b8aaa1be-6f0b-451f-9486-f631143c55ac" containerID="1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6" exitCode=0
Oct 30 17:25:17.217119 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.217096    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf" podUID="6ecefbcb-2750-4aba-ab53-12d644a896f5" containerName="maas-api" containerID="cri-o://27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7" gracePeriod=30
Oct 30 17:25:17.217327 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.217305    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"
Oct 30 17:25:17.217535 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.217503    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5" event={"ID":"b8aaa1be-6f0b-451f-9486-f631143c55ac","Type":"ContainerDied","Data":"1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6"}
Oct 30 17:25:17.217616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.217583    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5" event={"ID":"b8aaa1be-6f0b-451f-9486-f631143c55ac","Type":"ContainerDied","Data":"4483690b0d02d3048b8f27845365ca476a0703d971ddd2b910dc0262261b36cc"}
Oct 30 17:25:17.227870 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.227848    4497 scope.go:122] "RemoveContainer" containerID="1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6"
Oct 30 17:25:17.299913 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.299885    4497 scope.go:122] "RemoveContainer" containerID="1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6"
Oct 30 17:25:17.300277 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:25:17.300254    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6\": container with ID starting with 1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6 not found: ID does not exist" containerID="1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6"
Oct 30 17:25:17.300367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.300288    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6"} err="failed to get container status \"1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6\": rpc error: code = NotFound desc = could not find container \"1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6\": container with ID starting with 1acdb9a78a53376765a409e2db6a6b2ec247bf895e82ab1f6eea6f63a9ccdff6 not found: ID does not exist"
Oct 30 17:25:17.303609 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.303586    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"]
Oct 30 17:25:17.307538 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.307518    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bbc36f-744b8c8ff5-hsdf7"]
Oct 30 17:25:17.337473 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.337447    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"]
Oct 30 17:25:17.341400 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.341375    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2589b5-5f7f56b59c-jmpx5"]
Oct 30 17:25:17.548152 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.548127    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:25:17.581411 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.581383    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-maas-api-tls\") pod \"6ecefbcb-2750-4aba-ab53-12d644a896f5\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") "
Oct 30 17:25:17.581553 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.581498    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecefbcb-2750-4aba-ab53-12d644a896f5-kube-api-access-2v2gx\" (UniqueName: \"kubernetes.io/projected/6ecefbcb-2750-4aba-ab53-12d644a896f5-kube-api-access-2v2gx\") pod \"6ecefbcb-2750-4aba-ab53-12d644a896f5\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") "
Oct 30 17:25:17.581620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.581556    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-metrics-certs\") pod \"6ecefbcb-2750-4aba-ab53-12d644a896f5\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") "
Oct 30 17:25:17.581620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.581589    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ecefbcb-2750-4aba-ab53-12d644a896f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecefbcb-2750-4aba-ab53-12d644a896f5-openshift-service-ca\") pod \"6ecefbcb-2750-4aba-ab53-12d644a896f5\" (UID: \"6ecefbcb-2750-4aba-ab53-12d644a896f5\") "
Oct 30 17:25:17.582079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.582045    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecefbcb-2750-4aba-ab53-12d644a896f5-openshift-service-ca" pod "6ecefbcb-2750-4aba-ab53-12d644a896f5" (UID: "6ecefbcb-2750-4aba-ab53-12d644a896f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:25:17.583933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.583901    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-maas-api-tls" pod "6ecefbcb-2750-4aba-ab53-12d644a896f5" (UID: "6ecefbcb-2750-4aba-ab53-12d644a896f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:17.584043 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.583945    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecefbcb-2750-4aba-ab53-12d644a896f5-kube-api-access-2v2gx" pod "6ecefbcb-2750-4aba-ab53-12d644a896f5" (UID: "6ecefbcb-2750-4aba-ab53-12d644a896f5"). InnerVolumeSpecName "kube-api-access-2v2gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:17.584043 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.584004    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-metrics-certs" pod "6ecefbcb-2750-4aba-ab53-12d644a896f5" (UID: "6ecefbcb-2750-4aba-ab53-12d644a896f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:17.684020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.683955    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v2gx\" (UniqueName: \"kubernetes.io/projected/6ecefbcb-2750-4aba-ab53-12d644a896f5-kube-api-access-2v2gx\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:17.684020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.684019    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:17.684254 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.684033    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecefbcb-2750-4aba-ab53-12d644a896f5-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:17.684254 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:17.684048    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecefbcb-2750-4aba-ab53-12d644a896f5-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:18.224269 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.224238    4497 generic.go:356] "Generic (PLEG): container finished" podID="6ecefbcb-2750-4aba-ab53-12d644a896f5" containerID="27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7" exitCode=0
Oct 30 17:25:18.224750 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.224287    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"
Oct 30 17:25:18.224750 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.224323    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf" event={"ID":"6ecefbcb-2750-4aba-ab53-12d644a896f5","Type":"ContainerDied","Data":"27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7"}
Oct 30 17:25:18.224750 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.224385    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf" event={"ID":"6ecefbcb-2750-4aba-ab53-12d644a896f5","Type":"ContainerDied","Data":"a827ec6e94ca9a8e8ba412356c08be60f236c5469d24ec5ecb4d09ca3f63c665"}
Oct 30 17:25:18.224750 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.224414    4497 scope.go:122] "RemoveContainer" containerID="27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7"
Oct 30 17:25:18.234644 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.234626    4497 scope.go:122] "RemoveContainer" containerID="27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7"
Oct 30 17:25:18.234920 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:25:18.234898    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7\": container with ID starting with 27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7 not found: ID does not exist" containerID="27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7"
Oct 30 17:25:18.235019 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.234924    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7"} err="failed to get container status \"27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7\": rpc error: code = NotFound desc = could not find container \"27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7\": container with ID starting with 27b41af38f678e902352c33f2c43f19141bb9921fac217d3d3b6b9446bbfc3d7 not found: ID does not exist"
Oct 30 17:25:18.269056 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.269035    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"]
Oct 30 17:25:18.278010 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.277989    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c6596-9c7db9466-hgxvf"]
Oct 30 17:25:18.299986 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.299963    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecefbcb-2750-4aba-ab53-12d644a896f5" path="/var/lib/kubelet/pods/6ecefbcb-2750-4aba-ab53-12d644a896f5/volumes"
Oct 30 17:25:18.300394 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.300379    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8aaa1be-6f0b-451f-9486-f631143c55ac" path="/var/lib/kubelet/pods/b8aaa1be-6f0b-451f-9486-f631143c55ac/volumes"
Oct 30 17:25:18.300757 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:18.300745    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" path="/var/lib/kubelet/pods/ec157bad-0aa9-4abe-96fc-f1e7b7a4186e/volumes"
Oct 30 17:25:21.916137 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:25:21.916103    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:25:25.642976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:25.642940    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"]
Oct 30 17:25:25.645578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:25.643297    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy" containerID="cri-o://66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800" gracePeriod=30
Oct 30 17:25:30.923759 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:30.923705    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:25:31.045575 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045541    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istiod-ca-cert\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.045779 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045633    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-kube-api-access-nhq9r\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-kube-api-access-nhq9r\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.045779 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045694    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-token\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-token\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.045915 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045780    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-envoy\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.045915 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045813    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-socket\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.045915 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045885    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-podinfo\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.046079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045920    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-certs\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.046079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045972    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-data\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.046079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.045974    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istiod-ca-cert" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:25:31.046079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046015    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-credential-socket\") pod \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\" (UID: \"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0\") "
Oct 30 17:25:31.046301 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046152    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-socket" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:31.046520 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046488    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-credential-socket" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:31.046520 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046501    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-data" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:31.046682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046613    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.046682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046634    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.046682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046648    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.046682 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046663    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.046923 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.046687    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-certs" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:31.048086 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.048056    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-podinfo" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:25:31.048218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.048106    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-envoy" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:25:31.048395 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.048374    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-kube-api-access-nhq9r" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "kube-api-access-nhq9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:31.048464 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.048389    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-token" pod "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" (UID: "e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:31.147899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.147873    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.147899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.147896    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.148038 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.147905    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.148038 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.147914    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhq9r\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-kube-api-access-nhq9r\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.148038 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.147924    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:31.325628 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.325594    4497 generic.go:356] "Generic (PLEG): container finished" podID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerID="66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800" exitCode=0
Oct 30 17:25:31.325833 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.325652    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"
Oct 30 17:25:31.325833 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.325675    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" event={"ID":"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0","Type":"ContainerDied","Data":"66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800"}
Oct 30 17:25:31.325833 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.325749    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks" event={"ID":"e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0","Type":"ContainerDied","Data":"70c3f7c5b195898ff92b08568c18f86ab294a2a88b9c39dc6aecd2a1c18fae8b"}
Oct 30 17:25:31.325833 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.325763    4497 scope.go:122] "RemoveContainer" containerID="66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800"
Oct 30 17:25:31.338996 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.338978    4497 scope.go:122] "RemoveContainer" containerID="ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970"
Oct 30 17:25:31.347722 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.347705    4497 scope.go:122] "RemoveContainer" containerID="66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800"
Oct 30 17:25:31.347981 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:25:31.347965    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800\": container with ID starting with 66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800 not found: ID does not exist" containerID="66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800"
Oct 30 17:25:31.348051 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.347988    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800"} err="failed to get container status \"66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800\": rpc error: code = NotFound desc = could not find container \"66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800\": container with ID starting with 66562ffd5d522aa06c81f228cc2af8911073deff1807cfe7d3ba4acbe8769800 not found: ID does not exist"
Oct 30 17:25:31.348051 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.348005    4497 scope.go:122] "RemoveContainer" containerID="ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970"
Oct 30 17:25:31.348295 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:25:31.348276    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970\": container with ID starting with ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970 not found: ID does not exist" containerID="ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970"
Oct 30 17:25:31.348356 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.348300    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970"} err="failed to get container status \"ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970\": rpc error: code = NotFound desc = could not find container \"ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970\": container with ID starting with ce9680b003209657445d31f2169c12b5f692b3aeb6538e0845d300062aa0f970 not found: ID does not exist"
Oct 30 17:25:31.371590 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.371568    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"]
Oct 30 17:25:31.375612 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:31.375563    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-8lpks"]
Oct 30 17:25:32.300226 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:32.300180    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" path="/var/lib/kubelet/pods/e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0/volumes"
Oct 30 17:25:38.316607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:38.316570    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"]
Oct 30 17:25:43.317815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:43.317773    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9" podUID="62009120-1b30-4812-9e31-0aeaf65896a7" containerName="payload-pre-processing" containerID="cri-o://d61b09e19c53fc8d26095df6e7349bf98be2a94f73e2d44999e5fddd6b556cbf" gracePeriod=25
Oct 30 17:25:51.671036 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:51.670998    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9" podUID="62009120-1b30-4812-9e31-0aeaf65896a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:25:51.914484 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:25:51.914449    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:25:52.135268 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:52.135238    4497 scope.go:122] "RemoveContainer" containerID="856ffacb7068817509bc3fad08aa22c91038ae5ae279f6aef8b9cd9d40bf42e3"
Oct 30 17:25:57.984981 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:57.984948    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"]
Oct 30 17:25:57.985522 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:57.985169    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62" podUID="2cbbe2ac-a256-465f-a5b4-f80112cd862c" containerName="maas-api" containerID="cri-o://c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760" gracePeriod=30
Oct 30 17:25:58.275932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.275903    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:25:58.363406 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.363372    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbbe2ac-a256-465f-a5b4-f80112cd862c-kube-api-access-7f6f6\" (UniqueName: \"kubernetes.io/projected/2cbbe2ac-a256-465f-a5b4-f80112cd862c-kube-api-access-7f6f6\") pod \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") "
Oct 30 17:25:58.363573 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.363456    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbbe2ac-a256-465f-a5b4-f80112cd862c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbbe2ac-a256-465f-a5b4-f80112cd862c-openshift-service-ca\") pod \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") "
Oct 30 17:25:58.363617 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.363572    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-maas-api-tls\") pod \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") "
Oct 30 17:25:58.363656 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.363644    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-metrics-certs\") pod \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\" (UID: \"2cbbe2ac-a256-465f-a5b4-f80112cd862c\") "
Oct 30 17:25:58.363775 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.363757    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbbe2ac-a256-465f-a5b4-f80112cd862c-openshift-service-ca" pod "2cbbe2ac-a256-465f-a5b4-f80112cd862c" (UID: "2cbbe2ac-a256-465f-a5b4-f80112cd862c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:25:58.364251 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.364227    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cbbe2ac-a256-465f-a5b4-f80112cd862c-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:58.365661 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.365629    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-maas-api-tls" pod "2cbbe2ac-a256-465f-a5b4-f80112cd862c" (UID: "2cbbe2ac-a256-465f-a5b4-f80112cd862c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:58.365759 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.365658    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-metrics-certs" pod "2cbbe2ac-a256-465f-a5b4-f80112cd862c" (UID: "2cbbe2ac-a256-465f-a5b4-f80112cd862c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:58.365759 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.365704    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbbe2ac-a256-465f-a5b4-f80112cd862c-kube-api-access-7f6f6" pod "2cbbe2ac-a256-465f-a5b4-f80112cd862c" (UID: "2cbbe2ac-a256-465f-a5b4-f80112cd862c"). InnerVolumeSpecName "kube-api-access-7f6f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:58.376002 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.375979    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"]
Oct 30 17:25:58.376215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.376168    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x" podUID="f342d24d-33d2-4428-83e8-4d5d8eeee05d" containerName="maas-api" containerID="cri-o://159298503eace3d4c195973489987e3184877ae728daa7fcff04b9dbdae9d661" gracePeriod=30
Oct 30 17:25:58.465798 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.465761    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f6f6\" (UniqueName: \"kubernetes.io/projected/2cbbe2ac-a256-465f-a5b4-f80112cd862c-kube-api-access-7f6f6\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:58.465798 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.465791    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:58.465798 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.465799    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cbbe2ac-a256-465f-a5b4-f80112cd862c-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:58.508512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.508485    4497 generic.go:356] "Generic (PLEG): container finished" podID="f342d24d-33d2-4428-83e8-4d5d8eeee05d" containerID="159298503eace3d4c195973489987e3184877ae728daa7fcff04b9dbdae9d661" exitCode=0
Oct 30 17:25:58.508672 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.508557    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x" event={"ID":"f342d24d-33d2-4428-83e8-4d5d8eeee05d","Type":"ContainerDied","Data":"159298503eace3d4c195973489987e3184877ae728daa7fcff04b9dbdae9d661"}
Oct 30 17:25:58.509929 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.509906    4497 generic.go:356] "Generic (PLEG): container finished" podID="2cbbe2ac-a256-465f-a5b4-f80112cd862c" containerID="c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760" exitCode=0
Oct 30 17:25:58.510037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.509960    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"
Oct 30 17:25:58.510037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.509996    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62" event={"ID":"2cbbe2ac-a256-465f-a5b4-f80112cd862c","Type":"ContainerDied","Data":"c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760"}
Oct 30 17:25:58.510142 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.510042    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62" event={"ID":"2cbbe2ac-a256-465f-a5b4-f80112cd862c","Type":"ContainerDied","Data":"22da5f303d31c7761055edf6eefc96bb8209aaa791396436dd330d28c2cc804d"}
Oct 30 17:25:58.510142 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.510064    4497 scope.go:122] "RemoveContainer" containerID="c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760"
Oct 30 17:25:58.520657 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.520638    4497 scope.go:122] "RemoveContainer" containerID="c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760"
Oct 30 17:25:58.520925 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:25:58.520904    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760\": container with ID starting with c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760 not found: ID does not exist" containerID="c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760"
Oct 30 17:25:58.520984 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.520935    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760"} err="failed to get container status \"c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760\": rpc error: code = NotFound desc = could not find container \"c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760\": container with ID starting with c4e214699aa1c545e2d1823fef6c2543c3ab691daff07ad6abd2bf0229cd1760 not found: ID does not exist"
Oct 30 17:25:58.559011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.558982    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"]
Oct 30 17:25:58.561857 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.561834    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-3106d6-674d7c9fb6-ndp62"]
Oct 30 17:25:58.646646 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.627195    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:25:58.672627 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.672596    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-metrics-certs\") pod \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") "
Oct 30 17:25:58.672828 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.672644    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-maas-api-tls\") pod \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") "
Oct 30 17:25:58.672828 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.672771    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f342d24d-33d2-4428-83e8-4d5d8eeee05d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f342d24d-33d2-4428-83e8-4d5d8eeee05d-openshift-service-ca\") pod \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") "
Oct 30 17:25:58.672950 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.672846    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f342d24d-33d2-4428-83e8-4d5d8eeee05d-kube-api-access-76gwd\" (UniqueName: \"kubernetes.io/projected/f342d24d-33d2-4428-83e8-4d5d8eeee05d-kube-api-access-76gwd\") pod \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\" (UID: \"f342d24d-33d2-4428-83e8-4d5d8eeee05d\") "
Oct 30 17:25:58.674349 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.674309    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f342d24d-33d2-4428-83e8-4d5d8eeee05d-openshift-service-ca" pod "f342d24d-33d2-4428-83e8-4d5d8eeee05d" (UID: "f342d24d-33d2-4428-83e8-4d5d8eeee05d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:25:58.675392 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.675371    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f342d24d-33d2-4428-83e8-4d5d8eeee05d-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:58.680544 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.679276    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-maas-api-tls" pod "f342d24d-33d2-4428-83e8-4d5d8eeee05d" (UID: "f342d24d-33d2-4428-83e8-4d5d8eeee05d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:58.683693 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.683657    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-metrics-certs" pod "f342d24d-33d2-4428-83e8-4d5d8eeee05d" (UID: "f342d24d-33d2-4428-83e8-4d5d8eeee05d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:25:58.694552 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.694515    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f342d24d-33d2-4428-83e8-4d5d8eeee05d-kube-api-access-76gwd" pod "f342d24d-33d2-4428-83e8-4d5d8eeee05d" (UID: "f342d24d-33d2-4428-83e8-4d5d8eeee05d"). InnerVolumeSpecName "kube-api-access-76gwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:25:58.777013 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.776917    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:58.777013 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.776943    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f342d24d-33d2-4428-83e8-4d5d8eeee05d-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:58.777013 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:58.776953    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gwd\" (UniqueName: \"kubernetes.io/projected/f342d24d-33d2-4428-83e8-4d5d8eeee05d-kube-api-access-76gwd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:25:59.517037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:59.517003    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"
Oct 30 17:25:59.517037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:59.517016    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x" event={"ID":"f342d24d-33d2-4428-83e8-4d5d8eeee05d","Type":"ContainerDied","Data":"c5f5458f1cbdcf8e494252e9a64814e91faafd509dd2da44066ff50ae99e7c2f"}
Oct 30 17:25:59.517546 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:59.517078    4497 scope.go:122] "RemoveContainer" containerID="159298503eace3d4c195973489987e3184877ae728daa7fcff04b9dbdae9d661"
Oct 30 17:25:59.560625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:59.560598    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"]
Oct 30 17:25:59.569658 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:25:59.569631    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5c6b3c-9db74c6cf-znv2x"]
Oct 30 17:26:00.300345 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:00.300316    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbbe2ac-a256-465f-a5b4-f80112cd862c" path="/var/lib/kubelet/pods/2cbbe2ac-a256-465f-a5b4-f80112cd862c/volumes"
Oct 30 17:26:00.300793 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:00.300770    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f342d24d-33d2-4428-83e8-4d5d8eeee05d" path="/var/lib/kubelet/pods/f342d24d-33d2-4428-83e8-4d5d8eeee05d/volumes"
Oct 30 17:26:01.671117 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:01.671081    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9" podUID="62009120-1b30-4812-9e31-0aeaf65896a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:26:03.545665 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.545635    4497 generic.go:356] "Generic (PLEG): container finished" podID="62009120-1b30-4812-9e31-0aeaf65896a7" containerID="d61b09e19c53fc8d26095df6e7349bf98be2a94f73e2d44999e5fddd6b556cbf" exitCode=0
Oct 30 17:26:03.546021 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.545714    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9" event={"ID":"62009120-1b30-4812-9e31-0aeaf65896a7","Type":"ContainerDied","Data":"d61b09e19c53fc8d26095df6e7349bf98be2a94f73e2d44999e5fddd6b556cbf"}
Oct 30 17:26:03.570519 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.570499    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:26:03.638523 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.638432    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62009120-1b30-4812-9e31-0aeaf65896a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009120-1b30-4812-9e31-0aeaf65896a7-plugins-config-volume\") pod \"62009120-1b30-4812-9e31-0aeaf65896a7\" (UID: \"62009120-1b30-4812-9e31-0aeaf65896a7\") "
Oct 30 17:26:03.638523 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.638515    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62009120-1b30-4812-9e31-0aeaf65896a7-kube-api-access-6qpb2\" (UniqueName: \"kubernetes.io/projected/62009120-1b30-4812-9e31-0aeaf65896a7-kube-api-access-6qpb2\") pod \"62009120-1b30-4812-9e31-0aeaf65896a7\" (UID: \"62009120-1b30-4812-9e31-0aeaf65896a7\") "
Oct 30 17:26:03.638902 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.638875    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62009120-1b30-4812-9e31-0aeaf65896a7-plugins-config-volume" pod "62009120-1b30-4812-9e31-0aeaf65896a7" (UID: "62009120-1b30-4812-9e31-0aeaf65896a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:26:03.640836 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.640812    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62009120-1b30-4812-9e31-0aeaf65896a7-kube-api-access-6qpb2" pod "62009120-1b30-4812-9e31-0aeaf65896a7" (UID: "62009120-1b30-4812-9e31-0aeaf65896a7"). InnerVolumeSpecName "kube-api-access-6qpb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:03.739957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.739930    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62009120-1b30-4812-9e31-0aeaf65896a7-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:03.739957 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:03.739955    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qpb2\" (UniqueName: \"kubernetes.io/projected/62009120-1b30-4812-9e31-0aeaf65896a7-kube-api-access-6qpb2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:04.553807 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:04.553769    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9" event={"ID":"62009120-1b30-4812-9e31-0aeaf65896a7","Type":"ContainerDied","Data":"8c5552b4acb48a5f2b4094a937156b0728377cb27ce711c201d91c44e9c2b507"}
Oct 30 17:26:04.554223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:04.553819    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"
Oct 30 17:26:04.554223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:04.553849    4497 scope.go:122] "RemoveContainer" containerID="d61b09e19c53fc8d26095df6e7349bf98be2a94f73e2d44999e5fddd6b556cbf"
Oct 30 17:26:04.597501 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:04.597453    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"]
Oct 30 17:26:04.606689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:04.606660    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3106d6-7b54d898f5-g9zr9"]
Oct 30 17:26:06.299314 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:06.299272    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62009120-1b30-4812-9e31-0aeaf65896a7" path="/var/lib/kubelet/pods/62009120-1b30-4812-9e31-0aeaf65896a7/volumes"
Oct 30 17:26:06.569360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:06.569280    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"]
Oct 30 17:26:06.569894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:06.569852    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy" containerID="cri-o://9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999" gracePeriod=30
Oct 30 17:26:11.856036 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:11.855981    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:26:12.047978 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.047948    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-kube-api-access-vc9hn\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-kube-api-access-vc9hn\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.047978 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.047982    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istiod-ca-cert\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.048261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048032    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-podinfo\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.048261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048147    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-certs\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.048261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048178    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-credential-socket\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.048261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048235    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-data\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.048489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048267    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-envoy\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.048489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048342    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-token\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-token\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.048489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048454    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-socket\") pod \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\" (UID: \"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0\") "
Oct 30 17:26:12.048489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048460    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-certs" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:12.048709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048490    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-credential-socket" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:12.048709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048505    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istiod-ca-cert" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:26:12.048709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048695    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-data" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:12.049174 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048913    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-socket" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:12.049174 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.048993    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.049174 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.049009    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.049174 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.049022    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.049174 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.049036    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.050492 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.050471    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-token" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:12.050592 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.050581    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-kube-api-access-vc9hn" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "kube-api-access-vc9hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:12.050645 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.050632    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-podinfo" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:26:12.050700 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.050681    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-envoy" pod "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" (UID: "18e7bdc0-1950-4c66-96d6-30caa8f8c2c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:12.149784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.149719    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.149784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.149749    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.149784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.149758    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9hn\" (UniqueName: \"kubernetes.io/projected/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-kube-api-access-vc9hn\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.149784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.149767    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.149784 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.149776    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:12.607365 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.607330    4497 generic.go:356] "Generic (PLEG): container finished" podID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerID="9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999" exitCode=0
Oct 30 17:26:12.607365 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.607368    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" event={"ID":"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0","Type":"ContainerDied","Data":"9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999"}
Oct 30 17:26:12.607626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.607397    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"
Oct 30 17:26:12.607626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.607409    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9" event={"ID":"18e7bdc0-1950-4c66-96d6-30caa8f8c2c0","Type":"ContainerDied","Data":"893432a4f72365aa94fd2694fec66a4785f7082a2b8da087dc581c7338e7c354"}
Oct 30 17:26:12.607626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.607430    4497 scope.go:122] "RemoveContainer" containerID="9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999"
Oct 30 17:26:12.617006 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.616989    4497 scope.go:122] "RemoveContainer" containerID="19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15"
Oct 30 17:26:12.627831 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.627809    4497 scope.go:122] "RemoveContainer" containerID="9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999"
Oct 30 17:26:12.628115 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:12.628091    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999\": container with ID starting with 9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999 not found: ID does not exist" containerID="9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999"
Oct 30 17:26:12.628215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.628122    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999"} err="failed to get container status \"9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999\": rpc error: code = NotFound desc = could not find container \"9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999\": container with ID starting with 9c045bf1c2426930eb7f33c29db619442ec68c2e2ebcbfe05d044321271fb999 not found: ID does not exist"
Oct 30 17:26:12.628215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.628138    4497 scope.go:122] "RemoveContainer" containerID="19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15"
Oct 30 17:26:12.629134 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:12.629114    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15\": container with ID starting with 19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15 not found: ID does not exist" containerID="19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15"
Oct 30 17:26:12.629240 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.629169    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15"} err="failed to get container status \"19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15\": rpc error: code = NotFound desc = could not find container \"19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15\": container with ID starting with 19311b53ad46b9ebb92aa20eac6f9268cebf5efb02207efbb46eda7e31f97f15 not found: ID does not exist"
Oct 30 17:26:12.670442 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.670417    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"]
Oct 30 17:26:12.677588 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:12.677561    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-8rrs9"]
Oct 30 17:26:14.299960 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:14.299926    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" path="/var/lib/kubelet/pods/18e7bdc0-1950-4c66-96d6-30caa8f8c2c0/volumes"
Oct 30 17:26:18.422805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:18.422772    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"]
Oct 30 17:26:21.916300 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:21.916262    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:26:21.986445 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:21.986417    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"]
Oct 30 17:26:22.024149 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:22.024124    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"]
Oct 30 17:26:23.423727 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:23.423685    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds" podUID="45758dca-74d1-4578-8750-171b237c972c" containerName="payload-pre-processing" containerID="cri-o://b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2" gracePeriod=25
Oct 30 17:26:26.987668 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:26.987626    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5" podUID="74dc2710-b820-4b0e-92b2-ccf7022bacd7" containerName="payload-processing" containerID="cri-o://502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af" gracePeriod=25
Oct 30 17:26:27.025473 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:27.025410    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq" podUID="67793c9e-0130-43f7-817e-839d8dd97642" containerName="payload-pre-processing" containerID="cri-o://c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0" gracePeriod=25
Oct 30 17:26:29.353211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:29.353141    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds" podUID="45758dca-74d1-4578-8750-171b237c972c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:26:35.514408 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:35.514324    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq" podUID="67793c9e-0130-43f7-817e-839d8dd97642" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:26:35.514863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:35.514401    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5" podUID="74dc2710-b820-4b0e-92b2-ccf7022bacd7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:26:39.353360 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:39.353318    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds" podUID="45758dca-74d1-4578-8750-171b237c972c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:26:43.086943 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.086899    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"]
Oct 30 17:26:43.087527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.087221    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy" containerID="cri-o://43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b" gracePeriod=30
Oct 30 17:26:43.695856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.695793    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 17:26:43.796767 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.796732    4497 generic.go:356] "Generic (PLEG): container finished" podID="45758dca-74d1-4578-8750-171b237c972c" containerID="b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2" exitCode=0
Oct 30 17:26:43.796931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.796778    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"
Oct 30 17:26:43.796931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.796803    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds" event={"ID":"45758dca-74d1-4578-8750-171b237c972c","Type":"ContainerDied","Data":"b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2"}
Oct 30 17:26:43.796931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.796842    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds" event={"ID":"45758dca-74d1-4578-8750-171b237c972c","Type":"ContainerDied","Data":"73b05b78b19d62cef92a17e719923c7c185d127d1f3d9d277ee761b2ebab9d12"}
Oct 30 17:26:43.796931 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.796862    4497 scope.go:122] "RemoveContainer" containerID="b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2"
Oct 30 17:26:43.806057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.806034    4497 scope.go:122] "RemoveContainer" containerID="b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2"
Oct 30 17:26:43.806311 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:43.806291    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2\": container with ID starting with b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2 not found: ID does not exist" containerID="b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2"
Oct 30 17:26:43.806386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.806320    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2"} err="failed to get container status \"b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2\": rpc error: code = NotFound desc = could not find container \"b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2\": container with ID starting with b5d16cbb448916aac70dc1576566241cf963f8d9cda2c953dea124b7b6d4f0a2 not found: ID does not exist"
Oct 30 17:26:43.806702 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.806683    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45758dca-74d1-4578-8750-171b237c972c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45758dca-74d1-4578-8750-171b237c972c-plugins-config-volume\") pod \"45758dca-74d1-4578-8750-171b237c972c\" (UID: \"45758dca-74d1-4578-8750-171b237c972c\") "
Oct 30 17:26:43.806745 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.806730    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45758dca-74d1-4578-8750-171b237c972c-kube-api-access-gr7vx\" (UniqueName: \"kubernetes.io/projected/45758dca-74d1-4578-8750-171b237c972c-kube-api-access-gr7vx\") pod \"45758dca-74d1-4578-8750-171b237c972c\" (UID: \"45758dca-74d1-4578-8750-171b237c972c\") "
Oct 30 17:26:43.807029 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.807003    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45758dca-74d1-4578-8750-171b237c972c-plugins-config-volume" pod "45758dca-74d1-4578-8750-171b237c972c" (UID: "45758dca-74d1-4578-8750-171b237c972c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:26:43.807112 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.807079    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45758dca-74d1-4578-8750-171b237c972c-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:43.808705 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.808675    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45758dca-74d1-4578-8750-171b237c972c-kube-api-access-gr7vx" pod "45758dca-74d1-4578-8750-171b237c972c" (UID: "45758dca-74d1-4578-8750-171b237c972c"). InnerVolumeSpecName "kube-api-access-gr7vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:43.908802 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:43.908772    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr7vx\" (UniqueName: \"kubernetes.io/projected/45758dca-74d1-4578-8750-171b237c972c-kube-api-access-gr7vx\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:44.136746 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:44.136719    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"]
Oct 30 17:26:44.145914 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:44.145878    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-cb6251-59cbd5bd4c-f2zds"]
Oct 30 17:26:44.299580 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:44.299550    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45758dca-74d1-4578-8750-171b237c972c" path="/var/lib/kubelet/pods/45758dca-74d1-4578-8750-171b237c972c/volumes"
Oct 30 17:26:45.514341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:45.514295    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq" podUID="67793c9e-0130-43f7-817e-839d8dd97642" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:26:45.514341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:45.514320    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5" podUID="74dc2710-b820-4b0e-92b2-ccf7022bacd7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:26:46.724236 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:46.724194    4497 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:26:46.724634 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:46.724272    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:26:47.118160 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:47.118128    4497 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67793c9e_0130_43f7_817e_839d8dd97642.slice/crio-c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67793c9e_0130_43f7_817e_839d8dd97642.slice/crio-conmon-c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 17:26:47.310919 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.310896    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:26:47.314689 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.314669    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:26:47.448830 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.448800    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74dc2710-b820-4b0e-92b2-ccf7022bacd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc2710-b820-4b0e-92b2-ccf7022bacd7-plugins-config-volume\") pod \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\" (UID: \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\") "
Oct 30 17:26:47.449009 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.448864    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67793c9e-0130-43f7-817e-839d8dd97642-kube-api-access-4gw84\" (UniqueName: \"kubernetes.io/projected/67793c9e-0130-43f7-817e-839d8dd97642-kube-api-access-4gw84\") pod \"67793c9e-0130-43f7-817e-839d8dd97642\" (UID: \"67793c9e-0130-43f7-817e-839d8dd97642\") "
Oct 30 17:26:47.449009 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.448909    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67793c9e-0130-43f7-817e-839d8dd97642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67793c9e-0130-43f7-817e-839d8dd97642-plugins-config-volume\") pod \"67793c9e-0130-43f7-817e-839d8dd97642\" (UID: \"67793c9e-0130-43f7-817e-839d8dd97642\") "
Oct 30 17:26:47.449009 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.448989    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74dc2710-b820-4b0e-92b2-ccf7022bacd7-kube-api-access-gx4pc\" (UniqueName: \"kubernetes.io/projected/74dc2710-b820-4b0e-92b2-ccf7022bacd7-kube-api-access-gx4pc\") pod \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\" (UID: \"74dc2710-b820-4b0e-92b2-ccf7022bacd7\") "
Oct 30 17:26:47.449364 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.449335    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dc2710-b820-4b0e-92b2-ccf7022bacd7-plugins-config-volume" pod "74dc2710-b820-4b0e-92b2-ccf7022bacd7" (UID: "74dc2710-b820-4b0e-92b2-ccf7022bacd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:26:47.449494 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.449372    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67793c9e-0130-43f7-817e-839d8dd97642-plugins-config-volume" pod "67793c9e-0130-43f7-817e-839d8dd97642" (UID: "67793c9e-0130-43f7-817e-839d8dd97642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:26:47.449597 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.449581    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74dc2710-b820-4b0e-92b2-ccf7022bacd7-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:47.449668 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.449600    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67793c9e-0130-43f7-817e-839d8dd97642-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:47.451084 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.451062    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67793c9e-0130-43f7-817e-839d8dd97642-kube-api-access-4gw84" pod "67793c9e-0130-43f7-817e-839d8dd97642" (UID: "67793c9e-0130-43f7-817e-839d8dd97642"). InnerVolumeSpecName "kube-api-access-4gw84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:47.451493 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.451474    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dc2710-b820-4b0e-92b2-ccf7022bacd7-kube-api-access-gx4pc" pod "74dc2710-b820-4b0e-92b2-ccf7022bacd7" (UID: "74dc2710-b820-4b0e-92b2-ccf7022bacd7"). InnerVolumeSpecName "kube-api-access-gx4pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:47.550835 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.550757    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gw84\" (UniqueName: \"kubernetes.io/projected/67793c9e-0130-43f7-817e-839d8dd97642-kube-api-access-4gw84\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:47.550835 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.550788    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx4pc\" (UniqueName: \"kubernetes.io/projected/74dc2710-b820-4b0e-92b2-ccf7022bacd7-kube-api-access-gx4pc\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:47.821702 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.821616    4497 generic.go:356] "Generic (PLEG): container finished" podID="74dc2710-b820-4b0e-92b2-ccf7022bacd7" containerID="502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af" exitCode=0
Oct 30 17:26:47.821702 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.821664    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"
Oct 30 17:26:47.822173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.821699    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5" event={"ID":"74dc2710-b820-4b0e-92b2-ccf7022bacd7","Type":"ContainerDied","Data":"502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af"}
Oct 30 17:26:47.822173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.821778    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5" event={"ID":"74dc2710-b820-4b0e-92b2-ccf7022bacd7","Type":"ContainerDied","Data":"43e21598f6ff3069681cb444ab6ea29b9cc7359d2f839078682483f8801918af"}
Oct 30 17:26:47.822173 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.821810    4497 scope.go:122] "RemoveContainer" containerID="502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af"
Oct 30 17:26:47.823170 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.823146    4497 generic.go:356] "Generic (PLEG): container finished" podID="67793c9e-0130-43f7-817e-839d8dd97642" containerID="c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0" exitCode=0
Oct 30 17:26:47.823312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.823206    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"
Oct 30 17:26:47.823380 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.823208    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq" event={"ID":"67793c9e-0130-43f7-817e-839d8dd97642","Type":"ContainerDied","Data":"c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0"}
Oct 30 17:26:47.823441 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.823373    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq" event={"ID":"67793c9e-0130-43f7-817e-839d8dd97642","Type":"ContainerDied","Data":"092f628b8bd527150935a0c6e5fc6e55762849200291d69ffd0cc62d211dd365"}
Oct 30 17:26:47.831338 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.831322    4497 scope.go:122] "RemoveContainer" containerID="502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af"
Oct 30 17:26:47.831620 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:47.831601    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af\": container with ID starting with 502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af not found: ID does not exist" containerID="502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af"
Oct 30 17:26:47.831669 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.831626    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af"} err="failed to get container status \"502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af\": rpc error: code = NotFound desc = could not find container \"502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af\": container with ID starting with 502a8dc95ac5eabd912d94407d4e688bfc604155498c14ea7bc75e7e5ff4b2af not found: ID does not exist"
Oct 30 17:26:47.831669 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.831642    4497 scope.go:122] "RemoveContainer" containerID="c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0"
Oct 30 17:26:47.840369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.840349    4497 scope.go:122] "RemoveContainer" containerID="c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0"
Oct 30 17:26:47.840608 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:47.840590    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0\": container with ID starting with c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0 not found: ID does not exist" containerID="c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0"
Oct 30 17:26:47.840666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.840618    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0"} err="failed to get container status \"c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0\": rpc error: code = NotFound desc = could not find container \"c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0\": container with ID starting with c0884457241799e63a02db3dc746e882c2b804b6212ed66aec5e00bd097b4da0 not found: ID does not exist"
Oct 30 17:26:47.867718 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.867697    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"]
Oct 30 17:26:47.871981 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.871961    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-43197b-5cb45dd64c-t4nmq"]
Oct 30 17:26:47.900696 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.900651    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"]
Oct 30 17:26:47.906430 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:47.906405    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-43197b-5b989f75c8-7krk5"]
Oct 30 17:26:48.301808 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.301766    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67793c9e-0130-43f7-817e-839d8dd97642" path="/var/lib/kubelet/pods/67793c9e-0130-43f7-817e-839d8dd97642/volumes"
Oct 30 17:26:48.302211 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.302180    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74dc2710-b820-4b0e-92b2-ccf7022bacd7" path="/var/lib/kubelet/pods/74dc2710-b820-4b0e-92b2-ccf7022bacd7/volumes"
Oct 30 17:26:48.353773 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.353718    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 17:26:48.461172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461122    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-credential-socket\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.461172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461172    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-certs\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.461469 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461246    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-istio-token\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-istio-token\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.461469 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461419    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/999975c1-3ece-490c-ad9f-a039410f320a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999975c1-3ece-490c-ad9f-a039410f320a-istio-podinfo\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.461469 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461456    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-credential-socket" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:48.461667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461504    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999975c1-3ece-490c-ad9f-a039410f320a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999975c1-3ece-490c-ad9f-a039410f320a-istiod-ca-cert\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.461667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461650    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-kube-api-access-rx57g\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-kube-api-access-rx57g\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.461667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461655    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-certs" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:48.461849 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461702    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-data\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.461849 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461811    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-envoy\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.461982 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.461964    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-socket\") pod \"999975c1-3ece-490c-ad9f-a039410f320a\" (UID: \"999975c1-3ece-490c-ad9f-a039410f320a\") "
Oct 30 17:26:48.462070 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.462049    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999975c1-3ece-490c-ad9f-a039410f320a-istiod-ca-cert" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:26:48.462215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.462169    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-data" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:48.462215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.462201    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-socket" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:48.462535 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.462514    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999975c1-3ece-490c-ad9f-a039410f320a-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.462619 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.462538    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.462619 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.462551    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.462619 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.462563    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.462619 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.462578    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.464318 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.464297    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-kube-api-access-rx57g" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "kube-api-access-rx57g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:48.464469 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.464448    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/999975c1-3ece-490c-ad9f-a039410f320a-istio-podinfo" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:26:48.464572 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.464553    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-envoy" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:26:48.464626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.464600    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-istio-token" pod "999975c1-3ece-490c-ad9f-a039410f320a" (UID: "999975c1-3ece-490c-ad9f-a039410f320a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:26:48.563769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.563740    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.563769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.563766    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999975c1-3ece-490c-ad9f-a039410f320a-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.563769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.563775    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx57g\" (UniqueName: \"kubernetes.io/projected/999975c1-3ece-490c-ad9f-a039410f320a-kube-api-access-rx57g\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.564060 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.563784    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999975c1-3ece-490c-ad9f-a039410f320a-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:26:48.792685 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.792653    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"]
Oct 30 17:26:48.831792 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.831757    4497 generic.go:356] "Generic (PLEG): container finished" podID="999975c1-3ece-490c-ad9f-a039410f320a" containerID="43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b" exitCode=0
Oct 30 17:26:48.832303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.831807    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"
Oct 30 17:26:48.832303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.831836    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" event={"ID":"999975c1-3ece-490c-ad9f-a039410f320a","Type":"ContainerDied","Data":"43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b"}
Oct 30 17:26:48.832303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.831889    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms" event={"ID":"999975c1-3ece-490c-ad9f-a039410f320a","Type":"ContainerDied","Data":"5ee624eaa354ec21d29a692e5ad00d407fcd54e1bef8cecc12d4de65d1c19d29"}
Oct 30 17:26:48.832303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.831919    4497 scope.go:122] "RemoveContainer" containerID="43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b"
Oct 30 17:26:48.841852 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.841830    4497 scope.go:122] "RemoveContainer" containerID="2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7"
Oct 30 17:26:48.851411 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.851388    4497 scope.go:122] "RemoveContainer" containerID="43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b"
Oct 30 17:26:48.851856 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:48.851726    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b\": container with ID starting with 43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b not found: ID does not exist" containerID="43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b"
Oct 30 17:26:48.851856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.851765    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b"} err="failed to get container status \"43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b\": rpc error: code = NotFound desc = could not find container \"43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b\": container with ID starting with 43ad0fc3c161e61ca155253556a88cd05a3ebe0d37576eec4cd7572ad760c77b not found: ID does not exist"
Oct 30 17:26:48.851856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.851786    4497 scope.go:122] "RemoveContainer" containerID="2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7"
Oct 30 17:26:48.852101 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:48.852073    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7\": container with ID starting with 2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7 not found: ID does not exist" containerID="2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7"
Oct 30 17:26:48.852176 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.852112    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7"} err="failed to get container status \"2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7\": rpc error: code = NotFound desc = could not find container \"2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7\": container with ID starting with 2258c846b21c718ed4532a18221732ce6ee6e232b46b48c2578f6224a1cfdea7 not found: ID does not exist"
Oct 30 17:26:48.877666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.877629    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"]
Oct 30 17:26:48.887083 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:48.887059    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x2gms"]
Oct 30 17:26:50.299563 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:50.299528    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999975c1-3ece-490c-ad9f-a039410f320a" path="/var/lib/kubelet/pods/999975c1-3ece-490c-ad9f-a039410f320a/volumes"
Oct 30 17:26:51.915440 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:26:51.915392    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:26:53.793926 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:53.793880    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9" podUID="2f7bff6b-8665-4536-8e4a-796474a61f05" containerName="payload-pre-processing" containerID="cri-o://0d9e06d6a59559abc4eae3f12708a213c125d71b06165955e9afca0930f2289f" gracePeriod=25
Oct 30 17:26:55.120210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:55.120148    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"]
Oct 30 17:26:55.134691 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:55.134663    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"]
Oct 30 17:26:58.342663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:26:58.342628    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9" podUID="2f7bff6b-8665-4536-8e4a-796474a61f05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:05.378490 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.378422    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"]
Oct 30 17:27:05.379068 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.378645    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s" podUID="2106064d-663d-4cfa-9c7d-84eca30a97e1" containerName="maas-api" containerID="cri-o://a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55" gracePeriod=30
Oct 30 17:27:05.649785 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.649708    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 17:27:05.754907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.754877    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-metrics-certs\") pod \"2106064d-663d-4cfa-9c7d-84eca30a97e1\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") "
Oct 30 17:27:05.755074 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.754946    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-maas-api-tls\") pod \"2106064d-663d-4cfa-9c7d-84eca30a97e1\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") "
Oct 30 17:27:05.755074 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.755019    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2106064d-663d-4cfa-9c7d-84eca30a97e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2106064d-663d-4cfa-9c7d-84eca30a97e1-openshift-service-ca\") pod \"2106064d-663d-4cfa-9c7d-84eca30a97e1\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") "
Oct 30 17:27:05.755074 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.755043    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2106064d-663d-4cfa-9c7d-84eca30a97e1-kube-api-access-cj97g\" (UniqueName: \"kubernetes.io/projected/2106064d-663d-4cfa-9c7d-84eca30a97e1-kube-api-access-cj97g\") pod \"2106064d-663d-4cfa-9c7d-84eca30a97e1\" (UID: \"2106064d-663d-4cfa-9c7d-84eca30a97e1\") "
Oct 30 17:27:05.755410 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.755372    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2106064d-663d-4cfa-9c7d-84eca30a97e1-openshift-service-ca" pod "2106064d-663d-4cfa-9c7d-84eca30a97e1" (UID: "2106064d-663d-4cfa-9c7d-84eca30a97e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:05.757165 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.757141    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2106064d-663d-4cfa-9c7d-84eca30a97e1-kube-api-access-cj97g" pod "2106064d-663d-4cfa-9c7d-84eca30a97e1" (UID: "2106064d-663d-4cfa-9c7d-84eca30a97e1"). InnerVolumeSpecName "kube-api-access-cj97g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:05.757165 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.757153    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-metrics-certs" pod "2106064d-663d-4cfa-9c7d-84eca30a97e1" (UID: "2106064d-663d-4cfa-9c7d-84eca30a97e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:05.757300 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.757174    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-maas-api-tls" pod "2106064d-663d-4cfa-9c7d-84eca30a97e1" (UID: "2106064d-663d-4cfa-9c7d-84eca30a97e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:05.856286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.856253    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj97g\" (UniqueName: \"kubernetes.io/projected/2106064d-663d-4cfa-9c7d-84eca30a97e1-kube-api-access-cj97g\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:05.856286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.856283    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:05.856286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.856294    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2106064d-663d-4cfa-9c7d-84eca30a97e1-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:05.856532 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.856302    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2106064d-663d-4cfa-9c7d-84eca30a97e1-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:05.934565 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.934487    4497 generic.go:356] "Generic (PLEG): container finished" podID="2106064d-663d-4cfa-9c7d-84eca30a97e1" containerID="a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55" exitCode=0
Oct 30 17:27:05.934565 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.934524    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s" event={"ID":"2106064d-663d-4cfa-9c7d-84eca30a97e1","Type":"ContainerDied","Data":"a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55"}
Oct 30 17:27:05.934747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.934579    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s" event={"ID":"2106064d-663d-4cfa-9c7d-84eca30a97e1","Type":"ContainerDied","Data":"c98051db0b70ded644e5a511ceff9ea921f7ef5937f2fd551319cb7d8b8d21cc"}
Oct 30 17:27:05.934747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.934536    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"
Oct 30 17:27:05.934747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.934601    4497 scope.go:122] "RemoveContainer" containerID="a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55"
Oct 30 17:27:05.944519 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.944504    4497 scope.go:122] "RemoveContainer" containerID="a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55"
Oct 30 17:27:05.944761 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:05.944740    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55\": container with ID starting with a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55 not found: ID does not exist" containerID="a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55"
Oct 30 17:27:05.944805 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.944769    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55"} err="failed to get container status \"a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55\": rpc error: code = NotFound desc = could not find container \"a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55\": container with ID starting with a3c9023c409a125610e59093cd3a9659a2cf144de3c3a0755469dd9e6281bf55 not found: ID does not exist"
Oct 30 17:27:05.978363 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.978341    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"]
Oct 30 17:27:05.982863 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:05.982710    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-deb721-85c677d7d4-d886s"]
Oct 30 17:27:06.299871 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:06.299840    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2106064d-663d-4cfa-9c7d-84eca30a97e1" path="/var/lib/kubelet/pods/2106064d-663d-4cfa-9c7d-84eca30a97e1/volumes"
Oct 30 17:27:08.342243 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:08.342182    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9" podUID="2f7bff6b-8665-4536-8e4a-796474a61f05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:10.140716 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.140679    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main" containerID="cri-o://f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd" gracePeriod=45
Oct 30 17:27:10.158473 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.158444    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" containerID="cri-o://b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e" gracePeriod=45
Oct 30 17:27:10.204230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.204180    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused"
Oct 30 17:27:10.420880 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.420856    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 17:27:10.447374 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.447351    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 17:27:10.507964 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.507888    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-dshm\") pod \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") "
Oct 30 17:27:10.508160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.507990    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tmp-dir\") pod \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") "
Oct 30 17:27:10.508160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508052    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-kube-api-access-xq4wd\" (UniqueName: \"kubernetes.io/projected/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-kube-api-access-xq4wd\") pod \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") "
Oct 30 17:27:10.508160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508107    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-home\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-home\") pod \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") "
Oct 30 17:27:10.508381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508278    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tmp-dir" pod "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" (UID: "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:10.508381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508286    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-model-cache\") pod \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") "
Oct 30 17:27:10.508381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508355    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tls-certs\" (UniqueName: \"kubernetes.io/secret/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tls-certs\") pod \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\" (UID: \"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f\") "
Oct 30 17:27:10.508542 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508470    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-home" pod "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" (UID: "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:10.508542 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508507    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-model-cache" pod "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" (UID: "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:10.508771 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508750    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.508771 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508770    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.508971 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.508781    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.516649 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.511923    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tls-certs" pod "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" (UID: "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:10.516649 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.512934    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-dshm" pod "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" (UID: "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:10.516649 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.515510    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-kube-api-access-xq4wd" pod "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" (UID: "af0b1899-ebc4-4ad1-bb16-7fec9d8b741f"). InnerVolumeSpecName "kube-api-access-xq4wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:10.609398 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609366    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9461cfdb-edd6-495d-a013-3a721850d1a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/9461cfdb-edd6-495d-a013-3a721850d1a2-tls-certs\") pod \"9461cfdb-edd6-495d-a013-3a721850d1a2\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") "
Oct 30 17:27:10.609579 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609461    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-dshm\") pod \"9461cfdb-edd6-495d-a013-3a721850d1a2\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") "
Oct 30 17:27:10.609579 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609503    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-home\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-home\") pod \"9461cfdb-edd6-495d-a013-3a721850d1a2\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") "
Oct 30 17:27:10.609579 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609545    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-model-cache\") pod \"9461cfdb-edd6-495d-a013-3a721850d1a2\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") "
Oct 30 17:27:10.609787 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609640    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-tmp-dir\") pod \"9461cfdb-edd6-495d-a013-3a721850d1a2\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") "
Oct 30 17:27:10.609787 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609688    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9461cfdb-edd6-495d-a013-3a721850d1a2-kube-api-access-d7qt2\" (UniqueName: \"kubernetes.io/projected/9461cfdb-edd6-495d-a013-3a721850d1a2-kube-api-access-d7qt2\") pod \"9461cfdb-edd6-495d-a013-3a721850d1a2\" (UID: \"9461cfdb-edd6-495d-a013-3a721850d1a2\") "
Oct 30 17:27:10.609890 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609849    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-home" pod "9461cfdb-edd6-495d-a013-3a721850d1a2" (UID: "9461cfdb-edd6-495d-a013-3a721850d1a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:10.609934 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609893    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-model-cache" pod "9461cfdb-edd6-495d-a013-3a721850d1a2" (UID: "9461cfdb-edd6-495d-a013-3a721850d1a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:10.609934 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.609909    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-tmp-dir" pod "9461cfdb-edd6-495d-a013-3a721850d1a2" (UID: "9461cfdb-edd6-495d-a013-3a721850d1a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:10.610178 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.610155    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.610178 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.610175    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.610370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.610201    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.610370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.610213    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.610370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.610222    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.610370 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.610231    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq4wd\" (UniqueName: \"kubernetes.io/projected/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f-kube-api-access-xq4wd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.611588 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.611565    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9461cfdb-edd6-495d-a013-3a721850d1a2-tls-certs" pod "9461cfdb-edd6-495d-a013-3a721850d1a2" (UID: "9461cfdb-edd6-495d-a013-3a721850d1a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:10.611678 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.611592    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-dshm" pod "9461cfdb-edd6-495d-a013-3a721850d1a2" (UID: "9461cfdb-edd6-495d-a013-3a721850d1a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:10.611678 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.611652    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9461cfdb-edd6-495d-a013-3a721850d1a2-kube-api-access-d7qt2" pod "9461cfdb-edd6-495d-a013-3a721850d1a2" (UID: "9461cfdb-edd6-495d-a013-3a721850d1a2"). InnerVolumeSpecName "kube-api-access-d7qt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:10.711628 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.711539    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7qt2\" (UniqueName: \"kubernetes.io/projected/9461cfdb-edd6-495d-a013-3a721850d1a2-kube-api-access-d7qt2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.711628 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.711570    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9461cfdb-edd6-495d-a013-3a721850d1a2-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.711628 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.711599    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9461cfdb-edd6-495d-a013-3a721850d1a2-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:10.967371 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.967286    4497 generic.go:356] "Generic (PLEG): container finished" podID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerID="f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd" exitCode=0
Oct 30 17:27:10.967371 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.967329    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"
Oct 30 17:27:10.967572 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.967375    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" event={"ID":"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f","Type":"ContainerDied","Data":"f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd"}
Oct 30 17:27:10.967572 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.967415    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj" event={"ID":"af0b1899-ebc4-4ad1-bb16-7fec9d8b741f","Type":"ContainerDied","Data":"4e0e088360c0799f23d0d26ec596fd01654045432ee8f27e0e7fb1cce2f28fe8"}
Oct 30 17:27:10.967572 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.967436    4497 scope.go:122] "RemoveContainer" containerID="f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd"
Oct 30 17:27:10.968899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.968877    4497 generic.go:356] "Generic (PLEG): container finished" podID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerID="b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e" exitCode=0
Oct 30 17:27:10.969011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.968933    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"
Oct 30 17:27:10.969011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.968955    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" event={"ID":"9461cfdb-edd6-495d-a013-3a721850d1a2","Type":"ContainerDied","Data":"b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e"}
Oct 30 17:27:10.969011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.969003    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn" event={"ID":"9461cfdb-edd6-495d-a013-3a721850d1a2","Type":"ContainerDied","Data":"a84f40e47a3b2424c41b215cfda6d81ad33b17308ffb8a18aeac726273591a3f"}
Oct 30 17:27:10.978528 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.978496    4497 scope.go:122] "RemoveContainer" containerID="f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd"
Oct 30 17:27:10.978759 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:10.978742    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd\": container with ID starting with f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd not found: ID does not exist" containerID="f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd"
Oct 30 17:27:10.978859 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.978767    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd"} err="failed to get container status \"f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd\": rpc error: code = NotFound desc = could not find container \"f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd\": container with ID starting with f4b17236f2f072c33feb31288e5d23cd26ac8a864138aa8e0ad405d345852dfd not found: ID does not exist"
Oct 30 17:27:10.978859 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.978784    4497 scope.go:122] "RemoveContainer" containerID="b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e"
Oct 30 17:27:10.986997 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.986975    4497 scope.go:122] "RemoveContainer" containerID="b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e"
Oct 30 17:27:10.987289 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:10.987260    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e\": container with ID starting with b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e not found: ID does not exist" containerID="b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e"
Oct 30 17:27:10.987367 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:10.987296    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e"} err="failed to get container status \"b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e\": rpc error: code = NotFound desc = could not find container \"b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e\": container with ID starting with b3638083d266e719e7c30af3e87ed432627b8dd8862f2eab3be9eb38cd9ced2e not found: ID does not exist"
Oct 30 17:27:11.012026 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.011999    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"]
Oct 30 17:27:11.021465 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.021442    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/premium-w4-420ff3-kserve-5cfcb7b54c-n5qjn"]
Oct 30 17:27:11.050916 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.050720    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"]
Oct 30 17:27:11.054411 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.054380    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-2-w4-420ff3-kserve-85bbfbf945-bjchj"]
Oct 30 17:27:11.591555 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.591346    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"]
Oct 30 17:27:11.591979 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.591568    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq" podUID="f4de5d87-7ec5-46ee-ba50-1fc96353620d" containerName="maas-api" containerID="cri-o://6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f" gracePeriod=30
Oct 30 17:27:11.838663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.838641    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 17:27:11.924720 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.924646    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-metrics-certs\") pod \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") "
Oct 30 17:27:11.924720 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.924684    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4de5d87-7ec5-46ee-ba50-1fc96353620d-kube-api-access-jzstm\" (UniqueName: \"kubernetes.io/projected/f4de5d87-7ec5-46ee-ba50-1fc96353620d-kube-api-access-jzstm\") pod \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") "
Oct 30 17:27:11.924917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.924729    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-maas-api-tls\") pod \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") "
Oct 30 17:27:11.924917 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.924819    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4de5d87-7ec5-46ee-ba50-1fc96353620d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de5d87-7ec5-46ee-ba50-1fc96353620d-openshift-service-ca\") pod \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\" (UID: \"f4de5d87-7ec5-46ee-ba50-1fc96353620d\") "
Oct 30 17:27:11.925228 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.925171    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4de5d87-7ec5-46ee-ba50-1fc96353620d-openshift-service-ca" pod "f4de5d87-7ec5-46ee-ba50-1fc96353620d" (UID: "f4de5d87-7ec5-46ee-ba50-1fc96353620d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:11.926907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.926878    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-maas-api-tls" pod "f4de5d87-7ec5-46ee-ba50-1fc96353620d" (UID: "f4de5d87-7ec5-46ee-ba50-1fc96353620d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:11.927004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.926912    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4de5d87-7ec5-46ee-ba50-1fc96353620d-kube-api-access-jzstm" pod "f4de5d87-7ec5-46ee-ba50-1fc96353620d" (UID: "f4de5d87-7ec5-46ee-ba50-1fc96353620d"). InnerVolumeSpecName "kube-api-access-jzstm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:11.927004 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.926913    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-metrics-certs" pod "f4de5d87-7ec5-46ee-ba50-1fc96353620d" (UID: "f4de5d87-7ec5-46ee-ba50-1fc96353620d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:11.977524 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.977497    4497 generic.go:356] "Generic (PLEG): container finished" podID="f4de5d87-7ec5-46ee-ba50-1fc96353620d" containerID="6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f" exitCode=0
Oct 30 17:27:11.977624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.977539    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"
Oct 30 17:27:11.977624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.977546    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq" event={"ID":"f4de5d87-7ec5-46ee-ba50-1fc96353620d","Type":"ContainerDied","Data":"6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f"}
Oct 30 17:27:11.977624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.977585    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq" event={"ID":"f4de5d87-7ec5-46ee-ba50-1fc96353620d","Type":"ContainerDied","Data":"ed19e6370bf411484e3605a50723273e5722de1df9b55e117bfbb289aa1e84ed"}
Oct 30 17:27:11.977624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.977604    4497 scope.go:122] "RemoveContainer" containerID="6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f"
Oct 30 17:27:11.986349 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.986333    4497 scope.go:122] "RemoveContainer" containerID="6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f"
Oct 30 17:27:11.986589 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:11.986571    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f\": container with ID starting with 6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f not found: ID does not exist" containerID="6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f"
Oct 30 17:27:11.986630 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:11.986601    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f"} err="failed to get container status \"6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f\": rpc error: code = NotFound desc = could not find container \"6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f\": container with ID starting with 6a97ba83ae0f5e782ce9f7b155e07d883c605f34d36fe19ecf731e0682a0772f not found: ID does not exist"
Oct 30 17:27:12.022667 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.022631    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"]
Oct 30 17:27:12.028130 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.026338    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-799c76-577dcf7747-pxhfq"]
Oct 30 17:27:12.028130 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.026674    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:12.028130 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.026692    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzstm\" (UniqueName: \"kubernetes.io/projected/f4de5d87-7ec5-46ee-ba50-1fc96353620d-kube-api-access-jzstm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:12.028130 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.026702    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4de5d87-7ec5-46ee-ba50-1fc96353620d-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:12.028130 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.026710    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de5d87-7ec5-46ee-ba50-1fc96353620d-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:12.299521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.299490    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" path="/var/lib/kubelet/pods/9461cfdb-edd6-495d-a013-3a721850d1a2/volumes"
Oct 30 17:27:12.299923 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.299910    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" path="/var/lib/kubelet/pods/af0b1899-ebc4-4ad1-bb16-7fec9d8b741f/volumes"
Oct 30 17:27:12.300313 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:12.300300    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4de5d87-7ec5-46ee-ba50-1fc96353620d" path="/var/lib/kubelet/pods/f4de5d87-7ec5-46ee-ba50-1fc96353620d/volumes"
Oct 30 17:27:13.990856 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:13.990826    4497 generic.go:356] "Generic (PLEG): container finished" podID="2f7bff6b-8665-4536-8e4a-796474a61f05" containerID="0d9e06d6a59559abc4eae3f12708a213c125d71b06165955e9afca0930f2289f" exitCode=0
Oct 30 17:27:13.991287 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:13.990892    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9" event={"ID":"2f7bff6b-8665-4536-8e4a-796474a61f05","Type":"ContainerDied","Data":"0d9e06d6a59559abc4eae3f12708a213c125d71b06165955e9afca0930f2289f"}
Oct 30 17:27:14.057082 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.057031    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:27:14.149384 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.149353    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7bff6b-8665-4536-8e4a-796474a61f05-kube-api-access-2c48g\" (UniqueName: \"kubernetes.io/projected/2f7bff6b-8665-4536-8e4a-796474a61f05-kube-api-access-2c48g\") pod \"2f7bff6b-8665-4536-8e4a-796474a61f05\" (UID: \"2f7bff6b-8665-4536-8e4a-796474a61f05\") "
Oct 30 17:27:14.149613 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.149597    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7bff6b-8665-4536-8e4a-796474a61f05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7bff6b-8665-4536-8e4a-796474a61f05-plugins-config-volume\") pod \"2f7bff6b-8665-4536-8e4a-796474a61f05\" (UID: \"2f7bff6b-8665-4536-8e4a-796474a61f05\") "
Oct 30 17:27:14.149930 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.149910    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7bff6b-8665-4536-8e4a-796474a61f05-plugins-config-volume" pod "2f7bff6b-8665-4536-8e4a-796474a61f05" (UID: "2f7bff6b-8665-4536-8e4a-796474a61f05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:14.151426 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.151406    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7bff6b-8665-4536-8e4a-796474a61f05-kube-api-access-2c48g" pod "2f7bff6b-8665-4536-8e4a-796474a61f05" (UID: "2f7bff6b-8665-4536-8e4a-796474a61f05"). InnerVolumeSpecName "kube-api-access-2c48g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:14.251027 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.250988    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7bff6b-8665-4536-8e4a-796474a61f05-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:14.251027 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.251022    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c48g\" (UniqueName: \"kubernetes.io/projected/2f7bff6b-8665-4536-8e4a-796474a61f05-kube-api-access-2c48g\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:14.997526 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.997490    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9" event={"ID":"2f7bff6b-8665-4536-8e4a-796474a61f05","Type":"ContainerDied","Data":"622231f285716b11ed17c577d16bf3ebfc130143acbaec4629ae10d0c42bdcf7"}
Oct 30 17:27:14.997973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.997546    4497 scope.go:122] "RemoveContainer" containerID="0d9e06d6a59559abc4eae3f12708a213c125d71b06165955e9afca0930f2289f"
Oct 30 17:27:14.997973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:14.997626    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"
Oct 30 17:27:15.042026 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:15.041997    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"]
Oct 30 17:27:15.046061 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:15.046035    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d80ebb-b6b99d4bc-mvlg9"]
Oct 30 17:27:16.305027 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:16.304989    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7bff6b-8665-4536-8e4a-796474a61f05" path="/var/lib/kubelet/pods/2f7bff6b-8665-4536-8e4a-796474a61f05/volumes"
Oct 30 17:27:19.411436 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:19.411394    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"]
Oct 30 17:27:19.444558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:19.444530    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"]
Oct 30 17:27:19.688468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:19.688377    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"]
Oct 30 17:27:19.998389 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:19.998296    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"]
Oct 30 17:27:19.998560 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:19.998537    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy" containerID="cri-o://d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8" gracePeriod=30
Oct 30 17:27:21.915036 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:21.914999    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:27:24.412482 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:24.412439    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9" podUID="0e622cd5-6f48-4de4-af56-afd3a47279c5" containerName="payload-processing" containerID="cri-o://99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447" gracePeriod=25
Oct 30 17:27:24.445672 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:24.445615    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl" podUID="2592437d-ef7a-48d5-a319-35fe7287197d" containerName="payload-pre-processing" containerID="cri-o://82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172" gracePeriod=25
Oct 30 17:27:25.269692 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.269636    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 17:27:25.374525 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374495    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-certs\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.374729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374539    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-token\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-token\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.374729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374578    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-socket\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.374729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374605    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-podinfo\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.374729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374665    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4242887-cbd0-4d9a-ae41-732a0eec8669-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4242887-cbd0-4d9a-ae41-732a0eec8669-istiod-ca-cert\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.374729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374709    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-credential-socket\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.375020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374738    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-kube-api-access-lddcw\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-kube-api-access-lddcw\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.375020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374773    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-certs" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:25.375020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374809    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-envoy\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.375020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374854    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-data\") pod \"d4242887-cbd0-4d9a-ae41-732a0eec8669\" (UID: \"d4242887-cbd0-4d9a-ae41-732a0eec8669\") "
Oct 30 17:27:25.375020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.374935    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-socket" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:25.375341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.375043    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-credential-socket" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:25.375341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.375123    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4242887-cbd0-4d9a-ae41-732a0eec8669-istiod-ca-cert" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:25.375341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.375280    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-data" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:25.375512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.375442    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:25.375512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.375456    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:25.375512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.375464    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:25.375512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.375472    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:25.375512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.375479    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4242887-cbd0-4d9a-ae41-732a0eec8669-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:25.376922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.376895    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-kube-api-access-lddcw" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "kube-api-access-lddcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:25.377057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.376988    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-token" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:25.377263 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.377242    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-podinfo" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:27:25.377373 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.377308    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-envoy" pod "d4242887-cbd0-4d9a-ae41-732a0eec8669" (UID: "d4242887-cbd0-4d9a-ae41-732a0eec8669"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:25.469607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.469568    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"]
Oct 30 17:27:25.476722 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.476700    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:25.476722 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.476722    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:25.476830 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.476731    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lddcw\" (UniqueName: \"kubernetes.io/projected/d4242887-cbd0-4d9a-ae41-732a0eec8669-kube-api-access-lddcw\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:25.476830 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:25.476743    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4242887-cbd0-4d9a-ae41-732a0eec8669-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:26.064303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.064263    4497 generic.go:356] "Generic (PLEG): container finished" podID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerID="d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8" exitCode=0
Oct 30 17:27:26.064303 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.064299    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" event={"ID":"d4242887-cbd0-4d9a-ae41-732a0eec8669","Type":"ContainerDied","Data":"d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8"}
Oct 30 17:27:26.064583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.064349    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr" event={"ID":"d4242887-cbd0-4d9a-ae41-732a0eec8669","Type":"ContainerDied","Data":"812f5b6c22dd4d320bbf5e32bb745749408abdb0a509e7fbf2e58a4119bc897b"}
Oct 30 17:27:26.064583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.064373    4497 scope.go:122] "RemoveContainer" containerID="d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8"
Oct 30 17:27:26.064583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.064281    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"
Oct 30 17:27:26.078123 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.078102    4497 scope.go:122] "RemoveContainer" containerID="66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302"
Oct 30 17:27:26.088318 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.088291    4497 scope.go:122] "RemoveContainer" containerID="d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8"
Oct 30 17:27:26.088607 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:26.088589    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8\": container with ID starting with d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8 not found: ID does not exist" containerID="d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8"
Oct 30 17:27:26.088657 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.088616    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8"} err="failed to get container status \"d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8\": rpc error: code = NotFound desc = could not find container \"d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8\": container with ID starting with d773ec4a2d2dfd7f0ec9472f0d04ce61bed0040ef0219978664bd579b139edc8 not found: ID does not exist"
Oct 30 17:27:26.088657 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.088638    4497 scope.go:122] "RemoveContainer" containerID="66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302"
Oct 30 17:27:26.088942 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:26.088921    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302\": container with ID starting with 66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302 not found: ID does not exist" containerID="66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302"
Oct 30 17:27:26.089010 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.088951    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302"} err="failed to get container status \"66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302\": rpc error: code = NotFound desc = could not find container \"66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302\": container with ID starting with 66cb5a2d30ae7eab68c3d70e48d242b6949de0402cb82e638424e36ac8372302 not found: ID does not exist"
Oct 30 17:27:26.109502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.109321    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"]
Oct 30 17:27:26.118468 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.118445    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-799c76-openshift-default-7bd56ccd57-dtqzr"]
Oct 30 17:27:26.299218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.299167    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" path="/var/lib/kubelet/pods/d4242887-cbd0-4d9a-ae41-732a0eec8669/volumes"
Oct 30 17:27:26.542471 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.542436    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl" podUID="2592437d-ef7a-48d5-a319-35fe7287197d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:26.542471 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:26.542461    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9" podUID="0e622cd5-6f48-4de4-af56-afd3a47279c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:30.470769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:30.470704    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb" podUID="7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" containerName="payload-pre-processing" containerID="cri-o://6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f" gracePeriod=25
Oct 30 17:27:34.708341 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:34.708297    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main" containerID="cri-o://28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448" gracePeriod=45
Oct 30 17:27:34.968374 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:34.968323    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 17:27:35.083062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083029    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea83974-ad90-4136-bce6-f0e4eecce09a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea83974-ad90-4136-bce6-f0e4eecce09a-tls-certs\") pod \"2ea83974-ad90-4136-bce6-f0e4eecce09a\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") "
Oct 30 17:27:35.083241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083109    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea83974-ad90-4136-bce6-f0e4eecce09a-kube-api-access-dnvst\" (UniqueName: \"kubernetes.io/projected/2ea83974-ad90-4136-bce6-f0e4eecce09a-kube-api-access-dnvst\") pod \"2ea83974-ad90-4136-bce6-f0e4eecce09a\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") "
Oct 30 17:27:35.083241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083172    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-model-cache\") pod \"2ea83974-ad90-4136-bce6-f0e4eecce09a\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") "
Oct 30 17:27:35.083366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083273    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-dshm\") pod \"2ea83974-ad90-4136-bce6-f0e4eecce09a\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") "
Oct 30 17:27:35.083366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083328    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-tmp-dir\") pod \"2ea83974-ad90-4136-bce6-f0e4eecce09a\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") "
Oct 30 17:27:35.083366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083358    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-home\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-home\") pod \"2ea83974-ad90-4136-bce6-f0e4eecce09a\" (UID: \"2ea83974-ad90-4136-bce6-f0e4eecce09a\") "
Oct 30 17:27:35.083553 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083507    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-model-cache" pod "2ea83974-ad90-4136-bce6-f0e4eecce09a" (UID: "2ea83974-ad90-4136-bce6-f0e4eecce09a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:35.083639 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083597    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-tmp-dir" pod "2ea83974-ad90-4136-bce6-f0e4eecce09a" (UID: "2ea83974-ad90-4136-bce6-f0e4eecce09a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:35.083815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083795    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-home" pod "2ea83974-ad90-4136-bce6-f0e4eecce09a" (UID: "2ea83974-ad90-4136-bce6-f0e4eecce09a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:35.083986 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083954    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:35.083986 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083979    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:35.084133 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.083992    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:35.085248 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.085224    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea83974-ad90-4136-bce6-f0e4eecce09a-kube-api-access-dnvst" pod "2ea83974-ad90-4136-bce6-f0e4eecce09a" (UID: "2ea83974-ad90-4136-bce6-f0e4eecce09a"). InnerVolumeSpecName "kube-api-access-dnvst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:35.085413 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.085330    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea83974-ad90-4136-bce6-f0e4eecce09a-tls-certs" pod "2ea83974-ad90-4136-bce6-f0e4eecce09a" (UID: "2ea83974-ad90-4136-bce6-f0e4eecce09a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:35.085413 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.085344    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-dshm" pod "2ea83974-ad90-4136-bce6-f0e4eecce09a" (UID: "2ea83974-ad90-4136-bce6-f0e4eecce09a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:35.131583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.131548    4497 generic.go:356] "Generic (PLEG): container finished" podID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerID="28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448" exitCode=0
Oct 30 17:27:35.131735 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.131614    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"
Oct 30 17:27:35.131735 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.131622    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" event={"ID":"2ea83974-ad90-4136-bce6-f0e4eecce09a","Type":"ContainerDied","Data":"28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448"}
Oct 30 17:27:35.131735 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.131660    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6" event={"ID":"2ea83974-ad90-4136-bce6-f0e4eecce09a","Type":"ContainerDied","Data":"8eaa17778690d0cb0da7a65e62561e32d9c888c91663e403647d74f381efd6b7"}
Oct 30 17:27:35.131735 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.131680    4497 scope.go:122] "RemoveContainer" containerID="28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448"
Oct 30 17:27:35.141411 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.141386    4497 scope.go:122] "RemoveContainer" containerID="28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448"
Oct 30 17:27:35.141710 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:35.141691    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448\": container with ID starting with 28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448 not found: ID does not exist" containerID="28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448"
Oct 30 17:27:35.141807 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.141715    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448"} err="failed to get container status \"28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448\": rpc error: code = NotFound desc = could not find container \"28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448\": container with ID starting with 28c6d0bd5d9e1e796883da67ca1bf530ab6c86bde6729ee914e950753749f448 not found: ID does not exist"
Oct 30 17:27:35.179432 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.179397    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"]
Oct 30 17:27:35.184681 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.184630    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-w2-18519c-kserve-79d5b86f96-894h6"]
Oct 30 17:27:35.185348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.185308    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea83974-ad90-4136-bce6-f0e4eecce09a-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:35.185348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.185341    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea83974-ad90-4136-bce6-f0e4eecce09a-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:35.185528 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:35.185356    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvst\" (UniqueName: \"kubernetes.io/projected/2ea83974-ad90-4136-bce6-f0e4eecce09a-kube-api-access-dnvst\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:36.247836 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.247799    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"]
Oct 30 17:27:36.248412 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.248026    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2" podUID="7800f323-b297-49bb-b271-cf1fc2ce4a27" containerName="maas-api" containerID="cri-o://6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7" gracePeriod=30
Oct 30 17:27:36.300581 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.300548    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" path="/var/lib/kubelet/pods/2ea83974-ad90-4136-bce6-f0e4eecce09a/volumes"
Oct 30 17:27:36.520594 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.520569    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 17:27:36.542842 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.542787    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9" podUID="0e622cd5-6f48-4de4-af56-afd3a47279c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:36.543495 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.543462    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl" podUID="2592437d-ef7a-48d5-a319-35fe7287197d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:36.599875 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.599844    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7800f323-b297-49bb-b271-cf1fc2ce4a27-kube-api-access-g9cwc\" (UniqueName: \"kubernetes.io/projected/7800f323-b297-49bb-b271-cf1fc2ce4a27-kube-api-access-g9cwc\") pod \"7800f323-b297-49bb-b271-cf1fc2ce4a27\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") "
Oct 30 17:27:36.600051 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.599935    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls\") pod \"7800f323-b297-49bb-b271-cf1fc2ce4a27\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") "
Oct 30 17:27:36.600051 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.599978    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-metrics-certs\") pod \"7800f323-b297-49bb-b271-cf1fc2ce4a27\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") "
Oct 30 17:27:36.600051 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.600000    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7800f323-b297-49bb-b271-cf1fc2ce4a27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7800f323-b297-49bb-b271-cf1fc2ce4a27-openshift-service-ca\") pod \"7800f323-b297-49bb-b271-cf1fc2ce4a27\" (UID: \"7800f323-b297-49bb-b271-cf1fc2ce4a27\") "
Oct 30 17:27:36.600449 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.600426    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7800f323-b297-49bb-b271-cf1fc2ce4a27-openshift-service-ca" pod "7800f323-b297-49bb-b271-cf1fc2ce4a27" (UID: "7800f323-b297-49bb-b271-cf1fc2ce4a27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:36.602260 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.602229    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7800f323-b297-49bb-b271-cf1fc2ce4a27-kube-api-access-g9cwc" pod "7800f323-b297-49bb-b271-cf1fc2ce4a27" (UID: "7800f323-b297-49bb-b271-cf1fc2ce4a27"). InnerVolumeSpecName "kube-api-access-g9cwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:36.602489 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.602467    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-metrics-certs" pod "7800f323-b297-49bb-b271-cf1fc2ce4a27" (UID: "7800f323-b297-49bb-b271-cf1fc2ce4a27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:36.602607 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.602481    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls" pod "7800f323-b297-49bb-b271-cf1fc2ce4a27" (UID: "7800f323-b297-49bb-b271-cf1fc2ce4a27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:36.701963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.701931    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9cwc\" (UniqueName: \"kubernetes.io/projected/7800f323-b297-49bb-b271-cf1fc2ce4a27-kube-api-access-g9cwc\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:36.701963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.701961    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:36.701963 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.701970    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7800f323-b297-49bb-b271-cf1fc2ce4a27-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:36.702333 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:36.701978    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7800f323-b297-49bb-b271-cf1fc2ce4a27-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:37.145182 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.145145    4497 generic.go:356] "Generic (PLEG): container finished" podID="7800f323-b297-49bb-b271-cf1fc2ce4a27" containerID="6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7" exitCode=0
Oct 30 17:27:37.145369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.145204    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"
Oct 30 17:27:37.145369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.145228    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2" event={"ID":"7800f323-b297-49bb-b271-cf1fc2ce4a27","Type":"ContainerDied","Data":"6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7"}
Oct 30 17:27:37.145369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.145285    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2" event={"ID":"7800f323-b297-49bb-b271-cf1fc2ce4a27","Type":"ContainerDied","Data":"25f93e0e22f1e13fd7da12f48c811cb6901cdd5fe4f2be5aabce042b11ae4e80"}
Oct 30 17:27:37.145369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.145314    4497 scope.go:122] "RemoveContainer" containerID="6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7"
Oct 30 17:27:37.155422 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.155406    4497 scope.go:122] "RemoveContainer" containerID="6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7"
Oct 30 17:27:37.155670 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:37.155651    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7\": container with ID starting with 6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7 not found: ID does not exist" containerID="6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7"
Oct 30 17:27:37.155713 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.155679    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7"} err="failed to get container status \"6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7\": rpc error: code = NotFound desc = could not find container \"6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7\": container with ID starting with 6e6a1628601f249a2e8d50058d96fc479a15c1da50bacf13064af5ef602dbba7 not found: ID does not exist"
Oct 30 17:27:37.220011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.219982    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"]
Oct 30 17:27:37.229043 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:37.229015    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-420ff3-5dd79c86d8-ht4p2"]
Oct 30 17:27:38.298937 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:38.298903    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7800f323-b297-49bb-b271-cf1fc2ce4a27" path="/var/lib/kubelet/pods/7800f323-b297-49bb-b271-cf1fc2ce4a27/volumes"
Oct 30 17:27:38.322039 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:38.322010    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb" podUID="7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:41.627769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.627713    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"]
Oct 30 17:27:41.628289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628224    4497 cpu_manager.go:414] "containerMap: removing container" podUID="066e972e-b5f4-4778-985e-e4a3174f4bfb" containerName="payload-processing"
Oct 30 17:27:41.628289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628236    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066e972e-b5f4-4778-985e-e4a3174f4bfb" containerName="payload-processing"
Oct 30 17:27:41.628289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628250    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e90987f2-2ff8-49cb-a3a9-d4e398ee1928" containerName="payload-processing"
Oct 30 17:27:41.628289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628255    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90987f2-2ff8-49cb-a3a9-d4e398ee1928" containerName="payload-processing"
Oct 30 17:27:41.628289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628268    4497 cpu_manager.go:414] "containerMap: removing container" podUID="72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" containerName="payload-pre-processing"
Oct 30 17:27:41.628289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628276    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" containerName="payload-pre-processing"
Oct 30 17:27:41.628289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628286    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5a95d1b9-44a3-4292-a1bf-b2e974fffe81" containerName="payload-pre-processing"
Oct 30 17:27:41.628289 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628291    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a95d1b9-44a3-4292-a1bf-b2e974fffe81" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628304    4497 cpu_manager.go:414] "containerMap: removing container" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628309    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628315    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628319    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628325    4497 cpu_manager.go:414] "containerMap: removing container" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628329    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628335    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" containerName="maas-api"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628340    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" containerName="maas-api"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628348    4497 cpu_manager.go:414] "containerMap: removing container" podUID="153210bf-399a-41bd-a5e3-de2a38b7d549" containerName="maas-api"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628355    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153210bf-399a-41bd-a5e3-de2a38b7d549" containerName="maas-api"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628365    4497 cpu_manager.go:414] "containerMap: removing container" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628371    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628382    4497 cpu_manager.go:414] "containerMap: removing container" podUID="5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628388    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628399    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628405    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628416    4497 cpu_manager.go:414] "containerMap: removing container" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628422    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628430    4497 cpu_manager.go:414] "containerMap: removing container" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628437    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628445    4497 cpu_manager.go:414] "containerMap: removing container" podUID="525852cc-f693-4705-b159-d66720176b20" containerName="payload-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628451    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525852cc-f693-4705-b159-d66720176b20" containerName="payload-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628461    4497 cpu_manager.go:414] "containerMap: removing container" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628470    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628481    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3df30ece-9a53-4a35-be67-d8e6675d6ba8" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628487    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df30ece-9a53-4a35-be67-d8e6675d6ba8" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628497    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2f7bff6b-8665-4536-8e4a-796474a61f05" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628504    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7bff6b-8665-4536-8e4a-796474a61f05" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628517    4497 cpu_manager.go:414] "containerMap: removing container" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628524    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628536    4497 cpu_manager.go:414] "containerMap: removing container" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628543    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628555    4497 cpu_manager.go:414] "containerMap: removing container" podUID="a124b7a1-5f18-4779-a839-3c7a8c37c5f8" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628562    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a124b7a1-5f18-4779-a839-3c7a8c37c5f8" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628572    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628578    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628587    4497 cpu_manager.go:414] "containerMap: removing container" podUID="bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628594    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628609    4497 cpu_manager.go:414] "containerMap: removing container" podUID="eae7a56f-53f0-42f1-8430-ab5572dbeb4c" containerName="maas-api"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628614    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae7a56f-53f0-42f1-8430-ab5572dbeb4c" containerName="maas-api"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628624    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628630    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628638    4497 cpu_manager.go:414] "containerMap: removing container" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628644    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628654    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628660    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628668    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f7d643d6-6956-4378-aab5-b50f5f3cfc4a" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628674    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d643d6-6956-4378-aab5-b50f5f3cfc4a" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628688    4497 cpu_manager.go:414] "containerMap: removing container" podUID="67793c9e-0130-43f7-817e-839d8dd97642" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628695    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67793c9e-0130-43f7-817e-839d8dd97642" containerName="payload-pre-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628709    4497 cpu_manager.go:414] "containerMap: removing container" podUID="1151b05d-9e88-46f5-9e73-2c7b7b8e8829" containerName="payload-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628716    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1151b05d-9e88-46f5-9e73-2c7b7b8e8829" containerName="payload-processing"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628728    4497 cpu_manager.go:414] "containerMap: removing container" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628736    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy"
Oct 30 17:27:41.628709 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628746    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628753    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628766    4497 cpu_manager.go:414] "containerMap: removing container" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628773    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628783    4497 cpu_manager.go:414] "containerMap: removing container" podUID="48b6e4b9-63df-4089-86b2-7fdb181c1e78" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628790    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b6e4b9-63df-4089-86b2-7fdb181c1e78" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628801    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b1915fda-de7a-4552-a52d-469a2deb8ac2" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628808    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1915fda-de7a-4552-a52d-469a2deb8ac2" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628819    4497 cpu_manager.go:414] "containerMap: removing container" podUID="6ecefbcb-2750-4aba-ab53-12d644a896f5" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628826    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecefbcb-2750-4aba-ab53-12d644a896f5" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628843    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7800f323-b297-49bb-b271-cf1fc2ce4a27" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628849    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7800f323-b297-49bb-b271-cf1fc2ce4a27" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628858    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628865    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628874    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b8aaa1be-6f0b-451f-9486-f631143c55ac" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628881    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8aaa1be-6f0b-451f-9486-f631143c55ac" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628891    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628898    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628907    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2106064d-663d-4cfa-9c7d-84eca30a97e1" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628914    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2106064d-663d-4cfa-9c7d-84eca30a97e1" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628925    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f4de5d87-7ec5-46ee-ba50-1fc96353620d" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628932    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4de5d87-7ec5-46ee-ba50-1fc96353620d" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628943    4497 cpu_manager.go:414] "containerMap: removing container" podUID="62009120-1b30-4812-9e31-0aeaf65896a7" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628951    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62009120-1b30-4812-9e31-0aeaf65896a7" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628961    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628968    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628981    4497 cpu_manager.go:414] "containerMap: removing container" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.628988    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629001    4497 cpu_manager.go:414] "containerMap: removing container" podUID="be939ad3-346e-4401-a5d8-65d4f94d46aa" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629007    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be939ad3-346e-4401-a5d8-65d4f94d46aa" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629017    4497 cpu_manager.go:414] "containerMap: removing container" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629024    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629036    4497 cpu_manager.go:414] "containerMap: removing container" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629042    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629050    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f342d24d-33d2-4428-83e8-4d5d8eeee05d" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629058    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f342d24d-33d2-4428-83e8-4d5d8eeee05d" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629073    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2cbbe2ac-a256-465f-a5b4-f80112cd862c" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629080    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbbe2ac-a256-465f-a5b4-f80112cd862c" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629091    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629098    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629107    4497 cpu_manager.go:414] "containerMap: removing container" podUID="45758dca-74d1-4578-8750-171b237c972c" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629114    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45758dca-74d1-4578-8750-171b237c972c" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629129    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629135    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629145    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629151    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629161    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629169    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629179    4497 cpu_manager.go:414] "containerMap: removing container" podUID="bd864eb4-5923-49b8-abd6-c62b8b5e4420" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629203    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd864eb4-5923-49b8-abd6-c62b8b5e4420" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629218    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629225    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629234    4497 cpu_manager.go:414] "containerMap: removing container" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629240    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629251    4497 cpu_manager.go:414] "containerMap: removing container" podUID="157e4427-efba-47e6-8332-f60180eca846" containerName="discovery"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629257    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157e4427-efba-47e6-8332-f60180eca846" containerName="discovery"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629270    4497 cpu_manager.go:414] "containerMap: removing container" podUID="74dc2710-b820-4b0e-92b2-ccf7022bacd7" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629277    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dc2710-b820-4b0e-92b2-ccf7022bacd7" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629396    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cee62a7-517b-4a8b-8d59-0be8dd2c6d51" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629410    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3603e008-6a64-4524-9f75-f28a32a8f0f2" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629420    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629430    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629441    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="525852cc-f693-4705-b159-d66720176b20" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629452    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c06ac1a-0119-4dde-a02a-8ab491565a05" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629460    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df30ece-9a53-4a35-be67-d8e6675d6ba8" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629469    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7bff6b-8665-4536-8e4a-796474a61f05" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629483    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629493    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="9905c187-904a-44ee-ab84-f6e767fc887b" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629504    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="a124b7a1-5f18-4779-a839-3c7a8c37c5f8" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629513    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629520    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc660447-d7c5-4f66-8fd7-460e4b6f5f8b" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629529    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae7a56f-53f0-42f1-8430-ab5572dbeb4c" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629537    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629546    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629554    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0e40c4-79ba-44c1-b762-ab8dc252fbd9" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629561    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d643d6-6956-4378-aab5-b50f5f3cfc4a" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629569    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="67793c9e-0130-43f7-817e-839d8dd97642" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629579    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="1151b05d-9e88-46f5-9e73-2c7b7b8e8829" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629591    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="0191197b-1734-4678-9304-7bfff05b46ef" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629599    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629607    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629616    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b6e4b9-63df-4089-86b2-7fdb181c1e78" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629625    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1915fda-de7a-4552-a52d-469a2deb8ac2" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629633    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecefbcb-2750-4aba-ab53-12d644a896f5" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629644    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7800f323-b297-49bb-b271-cf1fc2ce4a27" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629653    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea83974-ad90-4136-bce6-f0e4eecce09a" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629660    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8aaa1be-6f0b-451f-9486-f631143c55ac" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629669    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff542801-6f7a-4588-9f8b-dac74cae73eb" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629678    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2106064d-663d-4cfa-9c7d-84eca30a97e1" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629686    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4de5d87-7ec5-46ee-ba50-1fc96353620d" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629694    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="62009120-1b30-4812-9e31-0aeaf65896a7" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629702    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3de812-692e-405a-953c-cc84de94107a" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629714    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b635ded-7d16-47e4-a0c6-a71d0a131533" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629722    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="be939ad3-346e-4401-a5d8-65d4f94d46aa" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629730    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a890f36-432f-4f13-b9c4-067c99f91990" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629745    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="9461cfdb-edd6-495d-a013-3a721850d1a2" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629752    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f342d24d-33d2-4428-83e8-4d5d8eeee05d" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629760    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbbe2ac-a256-465f-a5b4-f80112cd862c" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629776    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="45758dca-74d1-4578-8750-171b237c972c" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629786    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f917b48c-109a-4c9b-8a32-be33e53b320a" containerName="main"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629793    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629801    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd864eb4-5923-49b8-abd6-c62b8b5e4420" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629809    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95cc0cc-dbf7-4fc8-8fb7-9867a69b97e0" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629818    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c597a45-d66f-4231-8243-4867e0850f31" containerName="istio-proxy"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629827    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="157e4427-efba-47e6-8332-f60180eca846" containerName="discovery"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629836    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dc2710-b820-4b0e-92b2-ccf7022bacd7" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629845    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="066e972e-b5f4-4778-985e-e4a3174f4bfb" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629852    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90987f2-2ff8-49cb-a3a9-d4e398ee1928" containerName="payload-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629861    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fde66f-17f2-4ccf-bf68-c5f5b6cdfb59" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629869    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a95d1b9-44a3-4292-a1bf-b2e974fffe81" containerName="payload-pre-processing"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629879    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec157bad-0aa9-4abe-96fc-f1e7b7a4186e" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629887    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="153210bf-399a-41bd-a5e3-de2a38b7d549" containerName="maas-api"
Oct 30 17:27:41.630620 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.629896    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0b1899-ebc4-4ad1-bb16-7fec9d8b741f" containerName="main"
Oct 30 17:27:41.634932 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.634915    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.647154 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.647130    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-420ff3-openshift-default-dockercfg-mlfh4\""
Oct 30 17:27:41.659631 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.659598    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"]
Oct 30 17:27:41.753011 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.752978    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hw2t\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-kube-api-access-9hw2t\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.753218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.753025    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.753218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.753050    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.753218 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.753155    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.753416 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.753225    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.753416 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.753322    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.753416 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.753383    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.753578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.753430    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.753578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.753501    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854227 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854166    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hw2t\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-kube-api-access-9hw2t\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854422 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854254    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854422 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854291    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854422 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854352    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854422 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854390    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854427    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854481    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854507    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854568    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854865 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854676    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854865 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854795    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.854975 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.854913    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.855051 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.855030    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.855150 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.855134    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.857201 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.857163    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.857423 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.857406    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.874755 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.874728    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.875017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.874996    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hw2t\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-kube-api-access-9hw2t\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:41.948120 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:41.948038    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:42.267413 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:42.267382    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"]
Oct 30 17:27:42.273248 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:27:42.273221    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fac9f5_5b9c_47cf_9c53_710b21e23dac.slice/crio-aa6776bdbb7f1f6c9a3ec0ea42202933aca58e795d32be944a48412620ec3e3c WatchSource:0}: Error finding container aa6776bdbb7f1f6c9a3ec0ea42202933aca58e795d32be944a48412620ec3e3c: Status 404 returned error can't find the container with id aa6776bdbb7f1f6c9a3ec0ea42202933aca58e795d32be944a48412620ec3e3c
Oct 30 17:27:42.275915 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:42.275880    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:27:42.276030 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:42.275947    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:27:42.276030 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:42.275977    4497 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236080Ki","pods":"250"}
Oct 30 17:27:43.180108 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:43.180069    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" event={"ID":"f1fac9f5-5b9c-47cf-9c53-710b21e23dac","Type":"ContainerStarted","Data":"68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634"}
Oct 30 17:27:43.180484 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:43.180132    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" event={"ID":"f1fac9f5-5b9c-47cf-9c53-710b21e23dac","Type":"ContainerStarted","Data":"aa6776bdbb7f1f6c9a3ec0ea42202933aca58e795d32be944a48412620ec3e3c"}
Oct 30 17:27:43.232072 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:43.232008    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" podStartSLOduration=2.231988642 podStartE2EDuration="2.231988642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:43.229491547 +0000 UTC m=+3893.083369821" watchObservedRunningTime="2026-10-30 17:27:43.231988642 +0000 UTC m=+3893.085866907"
Oct 30 17:27:43.948456 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:43.948422    4497 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:43.949922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:43.949894    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:27:43.950055 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:43.949944    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" podUID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:27:44.752215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.752168    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:27:44.755756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.755736    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:27:44.906213 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.906147    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e622cd5-6f48-4de4-af56-afd3a47279c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e622cd5-6f48-4de4-af56-afd3a47279c5-plugins-config-volume\") pod \"0e622cd5-6f48-4de4-af56-afd3a47279c5\" (UID: \"0e622cd5-6f48-4de4-af56-afd3a47279c5\") "
Oct 30 17:27:44.906706 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.906681    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2592437d-ef7a-48d5-a319-35fe7287197d-kube-api-access-v9bhh\" (UniqueName: \"kubernetes.io/projected/2592437d-ef7a-48d5-a319-35fe7287197d-kube-api-access-v9bhh\") pod \"2592437d-ef7a-48d5-a319-35fe7287197d\" (UID: \"2592437d-ef7a-48d5-a319-35fe7287197d\") "
Oct 30 17:27:44.908434 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.908389    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e622cd5-6f48-4de4-af56-afd3a47279c5-plugins-config-volume" pod "0e622cd5-6f48-4de4-af56-afd3a47279c5" (UID: "0e622cd5-6f48-4de4-af56-afd3a47279c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:44.910209 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.908882    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e622cd5-6f48-4de4-af56-afd3a47279c5-kube-api-access-89w2l\" (UniqueName: \"kubernetes.io/projected/0e622cd5-6f48-4de4-af56-afd3a47279c5-kube-api-access-89w2l\") pod \"0e622cd5-6f48-4de4-af56-afd3a47279c5\" (UID: \"0e622cd5-6f48-4de4-af56-afd3a47279c5\") "
Oct 30 17:27:44.910325 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.910268    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2592437d-ef7a-48d5-a319-35fe7287197d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2592437d-ef7a-48d5-a319-35fe7287197d-plugins-config-volume\") pod \"2592437d-ef7a-48d5-a319-35fe7287197d\" (UID: \"2592437d-ef7a-48d5-a319-35fe7287197d\") "
Oct 30 17:27:44.911026 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.911006    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e622cd5-6f48-4de4-af56-afd3a47279c5-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:44.911475 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.911454    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2592437d-ef7a-48d5-a319-35fe7287197d-plugins-config-volume" pod "2592437d-ef7a-48d5-a319-35fe7287197d" (UID: "2592437d-ef7a-48d5-a319-35fe7287197d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:44.911779 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.911754    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2592437d-ef7a-48d5-a319-35fe7287197d-kube-api-access-v9bhh" pod "2592437d-ef7a-48d5-a319-35fe7287197d" (UID: "2592437d-ef7a-48d5-a319-35fe7287197d"). InnerVolumeSpecName "kube-api-access-v9bhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:44.913515 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.913495    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e622cd5-6f48-4de4-af56-afd3a47279c5-kube-api-access-89w2l" pod "0e622cd5-6f48-4de4-af56-afd3a47279c5" (UID: "0e622cd5-6f48-4de4-af56-afd3a47279c5"). InnerVolumeSpecName "kube-api-access-89w2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:44.919347 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.918850    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"]
Oct 30 17:27:44.948537 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.948501    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:27:44.948654 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:44.948561    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" podUID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:27:45.012446 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.012417    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bhh\" (UniqueName: \"kubernetes.io/projected/2592437d-ef7a-48d5-a319-35fe7287197d-kube-api-access-v9bhh\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:45.012446 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.012444    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89w2l\" (UniqueName: \"kubernetes.io/projected/0e622cd5-6f48-4de4-af56-afd3a47279c5-kube-api-access-89w2l\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:45.012617 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.012455    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2592437d-ef7a-48d5-a319-35fe7287197d-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:45.206198 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.206148    4497 generic.go:356] "Generic (PLEG): container finished" podID="0e622cd5-6f48-4de4-af56-afd3a47279c5" containerID="99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447" exitCode=0
Oct 30 17:27:45.206400 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.206238    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"
Oct 30 17:27:45.206400 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.206237    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9" event={"ID":"0e622cd5-6f48-4de4-af56-afd3a47279c5","Type":"ContainerDied","Data":"99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447"}
Oct 30 17:27:45.206400 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.206292    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9" event={"ID":"0e622cd5-6f48-4de4-af56-afd3a47279c5","Type":"ContainerDied","Data":"baf1f39c9ddd93eabb5050a72d0c30d6ad40f4b37837fdfee267f7cacf7b2824"}
Oct 30 17:27:45.206400 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.206314    4497 scope.go:122] "RemoveContainer" containerID="99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447"
Oct 30 17:27:45.207746 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.207725    4497 generic.go:356] "Generic (PLEG): container finished" podID="2592437d-ef7a-48d5-a319-35fe7287197d" containerID="82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172" exitCode=0
Oct 30 17:27:45.207850 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.207769    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl" event={"ID":"2592437d-ef7a-48d5-a319-35fe7287197d","Type":"ContainerDied","Data":"82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172"}
Oct 30 17:27:45.207850 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.207814    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl" event={"ID":"2592437d-ef7a-48d5-a319-35fe7287197d","Type":"ContainerDied","Data":"f192c31d58d95024a507cbcd8ab30e8c4b8c550b110457cdc22cb2247ef7dd82"}
Oct 30 17:27:45.207850 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.207781    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"
Oct 30 17:27:45.208117 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.208097    4497 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" secret="" err="secret \"e2e-worker-w4-420ff3-openshift-default-dockercfg-mlfh4\" not found"
Oct 30 17:27:45.216393 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.216347    4497 scope.go:122] "RemoveContainer" containerID="99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447"
Oct 30 17:27:45.216651 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:45.216631    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447\": container with ID starting with 99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447 not found: ID does not exist" containerID="99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447"
Oct 30 17:27:45.216704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.216658    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447"} err="failed to get container status \"99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447\": rpc error: code = NotFound desc = could not find container \"99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447\": container with ID starting with 99f8da937da2d67cd75c94b81fc595d4eb0a02d1ac417475b262c89f51e84447 not found: ID does not exist"
Oct 30 17:27:45.216704 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.216674    4497 scope.go:122] "RemoveContainer" containerID="82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172"
Oct 30 17:27:45.225500 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.225484    4497 scope.go:122] "RemoveContainer" containerID="82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172"
Oct 30 17:27:45.225708 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:45.225690    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172\": container with ID starting with 82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172 not found: ID does not exist" containerID="82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172"
Oct 30 17:27:45.225773 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.225714    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172"} err="failed to get container status \"82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172\": rpc error: code = NotFound desc = could not find container \"82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172\": container with ID starting with 82adf0159186d0e404dfea866135a6d11f6bda62f398ef5c41ebac3eb5a84172 not found: ID does not exist"
Oct 30 17:27:45.249598 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.249573    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"]
Oct 30 17:27:45.257650 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.257626    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b09d3-7d9f56ccc4-h4xb9"]
Oct 30 17:27:45.300270 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.298929    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"]
Oct 30 17:27:45.300270 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.298982    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0b09d3-85d48d4895-qqvhl"]
Oct 30 17:27:45.948666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.948629    4497 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 30 17:27:45.949079 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:45.948679    4497 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" podUID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 30 17:27:46.215474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:46.215386    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" podUID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" containerName="istio-proxy" containerID="cri-o://68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634" gracePeriod=30
Oct 30 17:27:46.303207 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:46.303145    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e622cd5-6f48-4de4-af56-afd3a47279c5" path="/var/lib/kubelet/pods/0e622cd5-6f48-4de4-af56-afd3a47279c5/volumes"
Oct 30 17:27:46.303616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:46.303600    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2592437d-ef7a-48d5-a319-35fe7287197d" path="/var/lib/kubelet/pods/2592437d-ef7a-48d5-a319-35fe7287197d/volumes"
Oct 30 17:27:46.560995 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:46.560961    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"]
Oct 30 17:27:46.561286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:46.561255    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt" podUID="918e1718-9153-44eb-8b8b-f26252675214" containerName="maas-api" containerID="cri-o://dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2" gracePeriod=30
Oct 30 17:27:46.864244 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:46.864221    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 17:27:47.033232 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.033164    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-maas-api-tls\") pod \"918e1718-9153-44eb-8b8b-f26252675214\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") "
Oct 30 17:27:47.033756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.033268    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-metrics-certs\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-metrics-certs\") pod \"918e1718-9153-44eb-8b8b-f26252675214\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") "
Oct 30 17:27:47.033756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.033418    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918e1718-9153-44eb-8b8b-f26252675214-kube-api-access-vm6wg\" (UniqueName: \"kubernetes.io/projected/918e1718-9153-44eb-8b8b-f26252675214-kube-api-access-vm6wg\") pod \"918e1718-9153-44eb-8b8b-f26252675214\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") "
Oct 30 17:27:47.033756 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.033494    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918e1718-9153-44eb-8b8b-f26252675214-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/918e1718-9153-44eb-8b8b-f26252675214-openshift-service-ca\") pod \"918e1718-9153-44eb-8b8b-f26252675214\" (UID: \"918e1718-9153-44eb-8b8b-f26252675214\") "
Oct 30 17:27:47.034651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.034528    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918e1718-9153-44eb-8b8b-f26252675214-openshift-service-ca" pod "918e1718-9153-44eb-8b8b-f26252675214" (UID: "918e1718-9153-44eb-8b8b-f26252675214"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:47.047571 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.047350    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918e1718-9153-44eb-8b8b-f26252675214-kube-api-access-vm6wg" pod "918e1718-9153-44eb-8b8b-f26252675214" (UID: "918e1718-9153-44eb-8b8b-f26252675214"). InnerVolumeSpecName "kube-api-access-vm6wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:47.047855 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.047827    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-metrics-certs" pod "918e1718-9153-44eb-8b8b-f26252675214" (UID: "918e1718-9153-44eb-8b8b-f26252675214"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:47.052369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.052326    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-maas-api-tls" pod "918e1718-9153-44eb-8b8b-f26252675214" (UID: "918e1718-9153-44eb-8b8b-f26252675214"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:27:47.134770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.134681    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/918e1718-9153-44eb-8b8b-f26252675214-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:47.134770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.134712    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:47.134770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.134722    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/918e1718-9153-44eb-8b8b-f26252675214-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:47.134770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.134732    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6wg\" (UniqueName: \"kubernetes.io/projected/918e1718-9153-44eb-8b8b-f26252675214-kube-api-access-vm6wg\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:47.222081 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.222045    4497 generic.go:356] "Generic (PLEG): container finished" podID="918e1718-9153-44eb-8b8b-f26252675214" containerID="dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2" exitCode=0
Oct 30 17:27:47.222290 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.222095    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"
Oct 30 17:27:47.222290 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.222129    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt" event={"ID":"918e1718-9153-44eb-8b8b-f26252675214","Type":"ContainerDied","Data":"dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2"}
Oct 30 17:27:47.222290 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.222208    4497 scope.go:122] "RemoveContainer" containerID="dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2"
Oct 30 17:27:47.222290 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.222208    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt" event={"ID":"918e1718-9153-44eb-8b8b-f26252675214","Type":"ContainerDied","Data":"b692d7ed26660902d658d63769c3fd33eabaa8a92e166e5d80429fa8e211188a"}
Oct 30 17:27:47.231685 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.231666    4497 scope.go:122] "RemoveContainer" containerID="dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2"
Oct 30 17:27:47.231956 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:47.231937    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2\": container with ID starting with dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2 not found: ID does not exist" containerID="dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2"
Oct 30 17:27:47.232059 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.231965    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2"} err="failed to get container status \"dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2\": rpc error: code = NotFound desc = could not find container \"dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2\": container with ID starting with dc6eecd07ef3193b85da3e4b5eee702ae322658d05a184d1c8c5c00f16eb0af2 not found: ID does not exist"
Oct 30 17:27:47.265259 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.265220    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"]
Oct 30 17:27:47.269049 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:47.269023    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-18519c-fc587ffcd-tcspt"]
Oct 30 17:27:48.302618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:48.302587    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918e1718-9153-44eb-8b8b-f26252675214" path="/var/lib/kubelet/pods/918e1718-9153-44eb-8b8b-f26252675214/volumes"
Oct 30 17:27:48.321571 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:48.321540    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb" podUID="7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:50.714080 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:50.714055    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 17:27:50.768985 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:50.768957    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-plugins-config-volume\") pod \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\" (UID: \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\") "
Oct 30 17:27:50.769140 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:50.769052    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-kube-api-access-9twtl\" (UniqueName: \"kubernetes.io/projected/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-kube-api-access-9twtl\") pod \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\" (UID: \"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300\") "
Oct 30 17:27:50.769402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:50.769379    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-plugins-config-volume" pod "7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" (UID: "7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:50.771148 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:50.771130    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-kube-api-access-9twtl" pod "7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" (UID: "7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300"). InnerVolumeSpecName "kube-api-access-9twtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:50.870959 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:50.870924    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:50.870959 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:50.870951    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9twtl\" (UniqueName: \"kubernetes.io/projected/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300-kube-api-access-9twtl\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:51.248062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.247985    4497 generic.go:356] "Generic (PLEG): container finished" podID="7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" containerID="6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f" exitCode=0
Oct 30 17:27:51.248062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.248037    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"
Oct 30 17:27:51.248062 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.248037    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb" event={"ID":"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300","Type":"ContainerDied","Data":"6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f"}
Oct 30 17:27:51.248358 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.248101    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb" event={"ID":"7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300","Type":"ContainerDied","Data":"3f509d86030d3be753f3ffb964d04b64b147183eb77a12a88a8bd33c36b1a085"}
Oct 30 17:27:51.248358 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.248143    4497 scope.go:122] "RemoveContainer" containerID="6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f"
Oct 30 17:27:51.258210 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.258175    4497 scope.go:122] "RemoveContainer" containerID="6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f"
Oct 30 17:27:51.258512 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:51.258494    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f\": container with ID starting with 6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f not found: ID does not exist" containerID="6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f"
Oct 30 17:27:51.258585 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.258523    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f"} err="failed to get container status \"6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f\": rpc error: code = NotFound desc = could not find container \"6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f\": container with ID starting with 6ff555b4e443eca1a394c49e44c9c81a605f3c370059db93e5cf05f8e57df01f not found: ID does not exist"
Oct 30 17:27:51.290737 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.290709    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"]
Oct 30 17:27:51.300392 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.300369    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-18519c-68bb968fcf-h2jkb"]
Oct 30 17:27:51.680276 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.680237    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"]
Oct 30 17:27:51.694679 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:51.694650    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"]
Oct 30 17:27:51.918888 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:51.918855    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:27:52.307911 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:52.307874    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" path="/var/lib/kubelet/pods/7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300/volumes"
Oct 30 17:27:54.975973 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:54.975948    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:55.014038 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014011    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-credential-socket\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014170 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014075    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-certs\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014170 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014102    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-kube-api-access-9hw2t\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-kube-api-access-9hw2t\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014170 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014117    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-data\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014170 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014137    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-envoy\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014405 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014227    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-token\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-token\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014405 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014253    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-podinfo\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014405 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014304    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istiod-ca-cert\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014405 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014316    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-credential-socket" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:55.014405 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014344    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-socket\") pod \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\" (UID: \"f1fac9f5-5b9c-47cf-9c53-710b21e23dac\") "
Oct 30 17:27:55.014666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014439    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-certs" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:55.014666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014492    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-data" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:55.014783 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014700    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istiod-ca-cert" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:27:55.014783 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014708    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-socket" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:55.014889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014839    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.014889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014857    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.014889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014871    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.014889 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014883    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.015106 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.014895    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.016314 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.016290    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-kube-api-access-9hw2t" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "kube-api-access-9hw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:55.016499 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.016483    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-token" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:27:55.016715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.016694    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-envoy" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:27:55.016781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.016734    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-podinfo" pod "f1fac9f5-5b9c-47cf-9c53-710b21e23dac" (UID: "f1fac9f5-5b9c-47cf-9c53-710b21e23dac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:27:55.115909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.115865    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.115909 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.115908    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.116171 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.115924    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hw2t\" (UniqueName: \"kubernetes.io/projected/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-kube-api-access-9hw2t\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.116171 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.115940    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1fac9f5-5b9c-47cf-9c53-710b21e23dac-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:27:55.131927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.131889    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"]
Oct 30 17:27:55.132221 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.132153    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy" containerID="cri-o://0dbba1838c49847dfa69ea85f5bbc2c3b5421bdbd2a6e4c00364627db05f7833" gracePeriod=30
Oct 30 17:27:55.271310 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.271219    4497 generic.go:356] "Generic (PLEG): container finished" podID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" containerID="68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634" exitCode=0
Oct 30 17:27:55.271310 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.271271    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"
Oct 30 17:27:55.271310 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.271278    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" event={"ID":"f1fac9f5-5b9c-47cf-9c53-710b21e23dac","Type":"ContainerDied","Data":"68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634"}
Oct 30 17:27:55.271585 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.271333    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh" event={"ID":"f1fac9f5-5b9c-47cf-9c53-710b21e23dac","Type":"ContainerDied","Data":"aa6776bdbb7f1f6c9a3ec0ea42202933aca58e795d32be944a48412620ec3e3c"}
Oct 30 17:27:55.271585 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.271370    4497 scope.go:122] "RemoveContainer" containerID="68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634"
Oct 30 17:27:55.284237 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.284217    4497 scope.go:122] "RemoveContainer" containerID="68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634"
Oct 30 17:27:55.284548 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:27:55.284525    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634\": container with ID starting with 68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634 not found: ID does not exist" containerID="68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634"
Oct 30 17:27:55.284649 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.284557    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634"} err="failed to get container status \"68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634\": rpc error: code = NotFound desc = could not find container \"68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634\": container with ID starting with 68c620d7aff79bfc87ac0f85a268de73ff219a966e5440eccdced0bbd042b634 not found: ID does not exist"
Oct 30 17:27:55.315238 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.315208    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"]
Oct 30 17:27:55.321729 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:55.321707    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-dgkzh"]
Oct 30 17:27:56.298715 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:56.298685    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" path="/var/lib/kubelet/pods/f1fac9f5-5b9c-47cf-9c53-710b21e23dac/volumes"
Oct 30 17:27:56.680875 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:56.680787    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp" podUID="c2eb9968-b523-4c4d-ae98-5183512a13ce" containerName="payload-processing" containerID="cri-o://0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51" gracePeriod=25
Oct 30 17:27:56.695823 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:56.695788    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l" podUID="ed713416-d816-4724-bfd3-88ad575d63f2" containerName="payload-pre-processing" containerID="cri-o://9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0" gracePeriod=25
Oct 30 17:27:58.321916 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:58.321874    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp" podUID="c2eb9968-b523-4c4d-ae98-5183512a13ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:27:59.329734 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:27:59.329697    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l" podUID="ed713416-d816-4724-bfd3-88ad575d63f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:28:00.299930 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.299794    4497 generic.go:356] "Generic (PLEG): container finished" podID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerID="0dbba1838c49847dfa69ea85f5bbc2c3b5421bdbd2a6e4c00364627db05f7833" exitCode=0
Oct 30 17:28:00.300229 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.300171    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" event={"ID":"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690","Type":"ContainerDied","Data":"0dbba1838c49847dfa69ea85f5bbc2c3b5421bdbd2a6e4c00364627db05f7833"}
Oct 30 17:28:00.386680 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.386653    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 17:28:00.470701 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.470610    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-certs\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.470701 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.470658    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-kube-api-access-jkwl2\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-kube-api-access-jkwl2\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.470944 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.470742    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-data\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.470944 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.470866    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-podinfo\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.470944 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.470926    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istiod-ca-cert\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.471080 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.470950    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-certs" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:00.471080 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.470998    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-credential-socket\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.471080 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471043    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-envoy\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.471263 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471100    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-socket\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.471263 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471136    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-token\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-token\") pod \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\" (UID: \"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690\") "
Oct 30 17:28:00.471388 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471331    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-credential-socket" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:00.471388 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471372    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istiod-ca-cert" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:00.471637 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471603    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-data" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:00.471637 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471612    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-socket" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:00.471815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471765    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:00.471815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471787    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:00.471815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471802    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:00.471815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471815    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:00.472033 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.471830    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:00.473477 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.473454    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-token" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:00.473591 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.473529    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-kube-api-access-jkwl2" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "kube-api-access-jkwl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:00.473591 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.473579    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-podinfo" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:28:00.473788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.473766    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-envoy" pod "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" (UID: "fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:00.572570 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.572541    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:00.572570 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.572566    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:00.572570 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.572575    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:00.572781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:00.572584    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkwl2\" (UniqueName: \"kubernetes.io/projected/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690-kube-api-access-jkwl2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:01.307550 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:01.307517    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"
Oct 30 17:28:01.307747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:01.307548    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z" event={"ID":"fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690","Type":"ContainerDied","Data":"de1bb7d93e20e92bfab2ad6ed2c68e2f17fba0c86926d5c5ec0e249f3fad260d"}
Oct 30 17:28:01.307747 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:01.307618    4497 scope.go:122] "RemoveContainer" containerID="0dbba1838c49847dfa69ea85f5bbc2c3b5421bdbd2a6e4c00364627db05f7833"
Oct 30 17:28:01.387045 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:01.387004    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"]
Oct 30 17:28:01.396471 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:01.396439    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-7b68z"]
Oct 30 17:28:02.298669 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:02.298591    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" path="/var/lib/kubelet/pods/fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690/volumes"
Oct 30 17:28:06.920579 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:06.920545    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"]
Oct 30 17:28:06.921030 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:06.920856    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy" containerID="cri-o://f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba" gracePeriod=30
Oct 30 17:28:08.322635 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:08.322599    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp" podUID="c2eb9968-b523-4c4d-ae98-5183512a13ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:28:09.329817 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:09.329778    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l" podUID="ed713416-d816-4724-bfd3-88ad575d63f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:28:09.892866 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:09.892824    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"]
Oct 30 17:28:09.893384 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:09.893339    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh" podUID="06ce2f6d-41de-4fbe-acd9-370f26fc69a3" containerName="maas-api" containerID="cri-o://e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02" gracePeriod=30
Oct 30 17:28:10.135647 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.135621    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 17:28:10.280866 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.280833    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-kube-api-access-vf4xr\" (UniqueName: \"kubernetes.io/projected/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-kube-api-access-vf4xr\") pod \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") "
Oct 30 17:28:10.281017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.280936    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-metrics-certs\") pod \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") "
Oct 30 17:28:10.281017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.280961    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-maas-api-tls\") pod \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") "
Oct 30 17:28:10.281017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.281002    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-openshift-service-ca\") pod \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\" (UID: \"06ce2f6d-41de-4fbe-acd9-370f26fc69a3\") "
Oct 30 17:28:10.281439 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.281411    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-openshift-service-ca" pod "06ce2f6d-41de-4fbe-acd9-370f26fc69a3" (UID: "06ce2f6d-41de-4fbe-acd9-370f26fc69a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:10.282991 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.282967    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-maas-api-tls" pod "06ce2f6d-41de-4fbe-acd9-370f26fc69a3" (UID: "06ce2f6d-41de-4fbe-acd9-370f26fc69a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:10.283122 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.283024    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-metrics-certs" pod "06ce2f6d-41de-4fbe-acd9-370f26fc69a3" (UID: "06ce2f6d-41de-4fbe-acd9-370f26fc69a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:10.283122 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.283085    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-kube-api-access-vf4xr" pod "06ce2f6d-41de-4fbe-acd9-370f26fc69a3" (UID: "06ce2f6d-41de-4fbe-acd9-370f26fc69a3"). InnerVolumeSpecName "kube-api-access-vf4xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:10.360176 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.360144    4497 generic.go:356] "Generic (PLEG): container finished" podID="06ce2f6d-41de-4fbe-acd9-370f26fc69a3" containerID="e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02" exitCode=0
Oct 30 17:28:10.360587 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.360206    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"
Oct 30 17:28:10.360587 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.360221    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh" event={"ID":"06ce2f6d-41de-4fbe-acd9-370f26fc69a3","Type":"ContainerDied","Data":"e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02"}
Oct 30 17:28:10.360587 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.360278    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh" event={"ID":"06ce2f6d-41de-4fbe-acd9-370f26fc69a3","Type":"ContainerDied","Data":"8940e00490536354145bd3baa7b65e991a09807dd708d72a3a7a8559efddca52"}
Oct 30 17:28:10.360587 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.360299    4497 scope.go:122] "RemoveContainer" containerID="e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02"
Oct 30 17:28:10.368895 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.368874    4497 scope.go:122] "RemoveContainer" containerID="e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02"
Oct 30 17:28:10.369210 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:28:10.369164    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02\": container with ID starting with e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02 not found: ID does not exist" containerID="e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02"
Oct 30 17:28:10.369273 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.369215    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02"} err="failed to get container status \"e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02\": rpc error: code = NotFound desc = could not find container \"e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02\": container with ID starting with e1c34360c06ab42c9cf56ca2565957ec72ca9392c2f81749de9c34e9892b3f02 not found: ID does not exist"
Oct 30 17:28:10.374406 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.374382    4497 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 17:28:10.374499 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.374436    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 17:28:10.382144 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.382123    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf4xr\" (UniqueName: \"kubernetes.io/projected/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-kube-api-access-vf4xr\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:10.382144 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.382145    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:10.382323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.382154    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:10.382323 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.382163    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ce2f6d-41de-4fbe-acd9-370f26fc69a3-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:10.403832 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.403810    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"]
Oct 30 17:28:10.410199 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:10.410164    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3503b-6558944b4-f4ghh"]
Oct 30 17:28:12.175241 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.175218    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:28:12.298521 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.298483    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ce2f6d-41de-4fbe-acd9-370f26fc69a3" path="/var/lib/kubelet/pods/06ce2f6d-41de-4fbe-acd9-370f26fc69a3/volumes"
Oct 30 17:28:12.303673 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303653    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-kube-api-access-ntm9b\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-kube-api-access-ntm9b\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.303767 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303683    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-credential-socket\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.303767 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303702    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-podinfo\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.303767 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303733    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-token\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-token\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.303922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303780    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-socket\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.303922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303799    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-envoy\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.303922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303815    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-data\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.303922 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303907    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639dc55b-adf9-4a59-afc0-29f509b002b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639dc55b-adf9-4a59-afc0-29f509b002b7-istiod-ca-cert\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.304128 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303936    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-credential-socket" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:12.304128 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.303966    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-certs\") pod \"639dc55b-adf9-4a59-afc0-29f509b002b7\" (UID: \"639dc55b-adf9-4a59-afc0-29f509b002b7\") "
Oct 30 17:28:12.304128 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.304100    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-socket" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:12.304344 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.304326    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.304402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.304341    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-certs" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:12.304402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.304361    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639dc55b-adf9-4a59-afc0-29f509b002b7-istiod-ca-cert" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:12.304402 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.304350    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.304512 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.304462    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-data" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:12.305956 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.305930    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-kube-api-access-ntm9b" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "kube-api-access-ntm9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:12.306040 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.305972    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-podinfo" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:28:12.306040 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.305969    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-envoy" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:28:12.306252 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.306239    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-token" pod "639dc55b-adf9-4a59-afc0-29f509b002b7" (UID: "639dc55b-adf9-4a59-afc0-29f509b002b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:12.372472 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.372431    4497 generic.go:356] "Generic (PLEG): container finished" podID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerID="f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba" exitCode=0
Oct 30 17:28:12.372627 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.372478    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" event={"ID":"639dc55b-adf9-4a59-afc0-29f509b002b7","Type":"ContainerDied","Data":"f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba"}
Oct 30 17:28:12.372627 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.372509    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"
Oct 30 17:28:12.372627 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.372532    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9" event={"ID":"639dc55b-adf9-4a59-afc0-29f509b002b7","Type":"ContainerDied","Data":"20699a5f614bae27341d6132a2548ea5233e92c1780af65ee1af9f06d72490d1"}
Oct 30 17:28:12.372627 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.372562    4497 scope.go:122] "RemoveContainer" containerID="f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba"
Oct 30 17:28:12.382090 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.382071    4497 scope.go:122] "RemoveContainer" containerID="be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00"
Oct 30 17:28:12.390714 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.390689    4497 scope.go:122] "RemoveContainer" containerID="f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba"
Oct 30 17:28:12.390987 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:28:12.390969    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba\": container with ID starting with f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba not found: ID does not exist" containerID="f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba"
Oct 30 17:28:12.391039 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.390997    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba"} err="failed to get container status \"f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba\": rpc error: code = NotFound desc = could not find container \"f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba\": container with ID starting with f27d7ca4428cbd3ce49dc5cceb5339eccaef33478da630457fba6254dc503fba not found: ID does not exist"
Oct 30 17:28:12.391039 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.391015    4497 scope.go:122] "RemoveContainer" containerID="be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00"
Oct 30 17:28:12.391257 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:28:12.391240    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00\": container with ID starting with be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00 not found: ID does not exist" containerID="be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00"
Oct 30 17:28:12.391302 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.391263    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00"} err="failed to get container status \"be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00\": rpc error: code = NotFound desc = could not find container \"be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00\": container with ID starting with be9a76b5b7223005e03d126445bf44f4dd5496eceafc4ccec1252f7c53338f00 not found: ID does not exist"
Oct 30 17:28:12.405457 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.405432    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntm9b\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-kube-api-access-ntm9b\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.405558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.405459    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.405558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.405472    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.405558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.405483    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.405558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.405494    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.405558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.405505    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639dc55b-adf9-4a59-afc0-29f509b002b7-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.405558 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.405516    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639dc55b-adf9-4a59-afc0-29f509b002b7-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:12.416459 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.416424    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"]
Oct 30 17:28:12.420739 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:12.420717    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-dq5j9"]
Oct 30 17:28:14.299493 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:14.299461    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" path="/var/lib/kubelet/pods/639dc55b-adf9-4a59-afc0-29f509b002b7/volumes"
Oct 30 17:28:16.965119 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:16.965094    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 17:28:16.993362 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:16.993341    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 17:28:17.163005 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.162979    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2eb9968-b523-4c4d-ae98-5183512a13ce-kube-api-access-jb4jk\" (UniqueName: \"kubernetes.io/projected/c2eb9968-b523-4c4d-ae98-5183512a13ce-kube-api-access-jb4jk\") pod \"c2eb9968-b523-4c4d-ae98-5183512a13ce\" (UID: \"c2eb9968-b523-4c4d-ae98-5183512a13ce\") "
Oct 30 17:28:17.163163 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.163088    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2eb9968-b523-4c4d-ae98-5183512a13ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9968-b523-4c4d-ae98-5183512a13ce-plugins-config-volume\") pod \"c2eb9968-b523-4c4d-ae98-5183512a13ce\" (UID: \"c2eb9968-b523-4c4d-ae98-5183512a13ce\") "
Oct 30 17:28:17.163163 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.163133    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed713416-d816-4724-bfd3-88ad575d63f2-kube-api-access-ngbv2\" (UniqueName: \"kubernetes.io/projected/ed713416-d816-4724-bfd3-88ad575d63f2-kube-api-access-ngbv2\") pod \"ed713416-d816-4724-bfd3-88ad575d63f2\" (UID: \"ed713416-d816-4724-bfd3-88ad575d63f2\") "
Oct 30 17:28:17.163163 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.163152    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed713416-d816-4724-bfd3-88ad575d63f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713416-d816-4724-bfd3-88ad575d63f2-plugins-config-volume\") pod \"ed713416-d816-4724-bfd3-88ad575d63f2\" (UID: \"ed713416-d816-4724-bfd3-88ad575d63f2\") "
Oct 30 17:28:17.163554 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.163521    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2eb9968-b523-4c4d-ae98-5183512a13ce-plugins-config-volume" pod "c2eb9968-b523-4c4d-ae98-5183512a13ce" (UID: "c2eb9968-b523-4c4d-ae98-5183512a13ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:17.163812 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.163783    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed713416-d816-4724-bfd3-88ad575d63f2-plugins-config-volume" pod "ed713416-d816-4724-bfd3-88ad575d63f2" (UID: "ed713416-d816-4724-bfd3-88ad575d63f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:17.165643 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.165609    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed713416-d816-4724-bfd3-88ad575d63f2-kube-api-access-ngbv2" pod "ed713416-d816-4724-bfd3-88ad575d63f2" (UID: "ed713416-d816-4724-bfd3-88ad575d63f2"). InnerVolumeSpecName "kube-api-access-ngbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:17.165739 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.165653    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2eb9968-b523-4c4d-ae98-5183512a13ce-kube-api-access-jb4jk" pod "c2eb9968-b523-4c4d-ae98-5183512a13ce" (UID: "c2eb9968-b523-4c4d-ae98-5183512a13ce"). InnerVolumeSpecName "kube-api-access-jb4jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:17.264904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.264873    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2eb9968-b523-4c4d-ae98-5183512a13ce-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:17.264904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.264900    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngbv2\" (UniqueName: \"kubernetes.io/projected/ed713416-d816-4724-bfd3-88ad575d63f2-kube-api-access-ngbv2\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:17.264904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.264909    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713416-d816-4724-bfd3-88ad575d63f2-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:17.265138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.264918    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4jk\" (UniqueName: \"kubernetes.io/projected/c2eb9968-b523-4c4d-ae98-5183512a13ce-kube-api-access-jb4jk\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:17.400212 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.400154    4497 generic.go:356] "Generic (PLEG): container finished" podID="c2eb9968-b523-4c4d-ae98-5183512a13ce" containerID="0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51" exitCode=0
Oct 30 17:28:17.400401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.400243    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"
Oct 30 17:28:17.400401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.400240    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp" event={"ID":"c2eb9968-b523-4c4d-ae98-5183512a13ce","Type":"ContainerDied","Data":"0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51"}
Oct 30 17:28:17.400401 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.400370    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp" event={"ID":"c2eb9968-b523-4c4d-ae98-5183512a13ce","Type":"ContainerDied","Data":"503512f063e32d3dfdb6892d1457c58baac072eafd846a36a09f1f272a4c14cd"}
Oct 30 17:28:17.400589 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.400403    4497 scope.go:122] "RemoveContainer" containerID="0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51"
Oct 30 17:28:17.401677 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.401651    4497 generic.go:356] "Generic (PLEG): container finished" podID="ed713416-d816-4724-bfd3-88ad575d63f2" containerID="9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0" exitCode=0
Oct 30 17:28:17.401809 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.401680    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l" event={"ID":"ed713416-d816-4724-bfd3-88ad575d63f2","Type":"ContainerDied","Data":"9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0"}
Oct 30 17:28:17.401809 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.401721    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"
Oct 30 17:28:17.401937 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.401725    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l" event={"ID":"ed713416-d816-4724-bfd3-88ad575d63f2","Type":"ContainerDied","Data":"aba76442fc168017ef5fb6a9faac68c2bd81f9ca8f0ffcbb572f690d23bf4360"}
Oct 30 17:28:17.412288 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.412270    4497 scope.go:122] "RemoveContainer" containerID="0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51"
Oct 30 17:28:17.413343 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:28:17.413320    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51\": container with ID starting with 0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51 not found: ID does not exist" containerID="0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51"
Oct 30 17:28:17.413513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.413483    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51"} err="failed to get container status \"0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51\": rpc error: code = NotFound desc = could not find container \"0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51\": container with ID starting with 0dac2e15d390c51c660f5aedaeccf941fe4575b7b830d079a033be0960306d51 not found: ID does not exist"
Oct 30 17:28:17.413513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.413514    4497 scope.go:122] "RemoveContainer" containerID="9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0"
Oct 30 17:28:17.423730 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.423669    4497 scope.go:122] "RemoveContainer" containerID="9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0"
Oct 30 17:28:17.424415 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:28:17.424396    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0\": container with ID starting with 9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0 not found: ID does not exist" containerID="9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0"
Oct 30 17:28:17.424502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.424420    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0"} err="failed to get container status \"9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0\": rpc error: code = NotFound desc = could not find container \"9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0\": container with ID starting with 9489fcce1f00f4b6a7fe2d059ffcc48ae26945681c54f670b9f434d0d3ca11e0 not found: ID does not exist"
Oct 30 17:28:17.456018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.455993    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"]
Oct 30 17:28:17.459136 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.459102    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3503b-765b7d95b4-lb6hp"]
Oct 30 17:28:17.487933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.487861    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"]
Oct 30 17:28:17.491637 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:17.491615    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3503b-c96746595-hdx4l"]
Oct 30 17:28:18.298825 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:18.298789    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2eb9968-b523-4c4d-ae98-5183512a13ce" path="/var/lib/kubelet/pods/c2eb9968-b523-4c4d-ae98-5183512a13ce/volumes"
Oct 30 17:28:18.299222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:18.299163    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed713416-d816-4724-bfd3-88ad575d63f2" path="/var/lib/kubelet/pods/ed713416-d816-4724-bfd3-88ad575d63f2/volumes"
Oct 30 17:28:19.494616 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.494582    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"]
Oct 30 17:28:19.495353 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.495318    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9" podUID="48c661b8-5a2a-4bb9-b12a-deb30e055e0b" containerName="maas-api" containerID="cri-o://2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9" gracePeriod=30
Oct 30 17:28:19.753515 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.753496    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:28:19.790078 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.790049    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-kube-api-access-dttpq\" (UniqueName: \"kubernetes.io/projected/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-kube-api-access-dttpq\") pod \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") "
Oct 30 17:28:19.790230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.790084    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-maas-api-tls\") pod \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") "
Oct 30 17:28:19.790230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.790109    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-openshift-service-ca\") pod \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") "
Oct 30 17:28:19.790230 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.790135    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-metrics-certs\") pod \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\" (UID: \"48c661b8-5a2a-4bb9-b12a-deb30e055e0b\") "
Oct 30 17:28:19.790507 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.790477    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-openshift-service-ca" pod "48c661b8-5a2a-4bb9-b12a-deb30e055e0b" (UID: "48c661b8-5a2a-4bb9-b12a-deb30e055e0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:28:19.792366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.792340    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-metrics-certs" pod "48c661b8-5a2a-4bb9-b12a-deb30e055e0b" (UID: "48c661b8-5a2a-4bb9-b12a-deb30e055e0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:19.792739 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.792712    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-maas-api-tls" pod "48c661b8-5a2a-4bb9-b12a-deb30e055e0b" (UID: "48c661b8-5a2a-4bb9-b12a-deb30e055e0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:28:19.792841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.792732    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-kube-api-access-dttpq" pod "48c661b8-5a2a-4bb9-b12a-deb30e055e0b" (UID: "48c661b8-5a2a-4bb9-b12a-deb30e055e0b"). InnerVolumeSpecName "kube-api-access-dttpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:28:19.891420 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.891387    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dttpq\" (UniqueName: \"kubernetes.io/projected/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-kube-api-access-dttpq\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:19.891420 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.891418    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:19.891618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.891427    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:19.891618 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:19.891437    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c661b8-5a2a-4bb9-b12a-deb30e055e0b-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:28:20.419513 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.419475    4497 generic.go:356] "Generic (PLEG): container finished" podID="48c661b8-5a2a-4bb9-b12a-deb30e055e0b" containerID="2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9" exitCode=0
Oct 30 17:28:20.419694 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.419523    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"
Oct 30 17:28:20.419694 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.419554    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9" event={"ID":"48c661b8-5a2a-4bb9-b12a-deb30e055e0b","Type":"ContainerDied","Data":"2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9"}
Oct 30 17:28:20.419694 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.419621    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9" event={"ID":"48c661b8-5a2a-4bb9-b12a-deb30e055e0b","Type":"ContainerDied","Data":"75643d0728b88106bff39e90af649a0c763061dd76c3fb4c854dfc016802eee5"}
Oct 30 17:28:20.419694 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.419649    4497 scope.go:122] "RemoveContainer" containerID="2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9"
Oct 30 17:28:20.428586 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.428567    4497 scope.go:122] "RemoveContainer" containerID="2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9"
Oct 30 17:28:20.428855 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:28:20.428834    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9\": container with ID starting with 2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9 not found: ID does not exist" containerID="2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9"
Oct 30 17:28:20.428927 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.428862    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9"} err="failed to get container status \"2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9\": rpc error: code = NotFound desc = could not find container \"2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9\": container with ID starting with 2a942fd45e5b1596eb52f96b5016f59b7feb50e62eb7faf04a7eaf47fa1355e9 not found: ID does not exist"
Oct 30 17:28:20.465530 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.465499    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"]
Oct 30 17:28:20.470059 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:20.470034    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-13c495-66cbdcf8c6-d9ss9"]
Oct 30 17:28:21.915304 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:28:21.915272    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:28:22.298231 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:22.298199    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c661b8-5a2a-4bb9-b12a-deb30e055e0b" path="/var/lib/kubelet/pods/48c661b8-5a2a-4bb9-b12a-deb30e055e0b/volumes"
Oct 30 17:28:38.804899 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:38.804860    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"]
Oct 30 17:28:43.805803 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:43.805745    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt" podUID="b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" containerName="payload-processing" containerID="cri-o://da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2" gracePeriod=25
Oct 30 17:28:44.686324 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:44.686284    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt" podUID="b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:28:51.914258 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:28:51.914174    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:28:54.686519 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:28:54.686482    4497 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt" podUID="b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 17:29:01.354348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.354308    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"]
Oct 30 17:29:01.354933 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.354550    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs" podUID="fa4050d9-892d-47e9-adf4-96a72db5b1a5" containerName="maas-api" containerID="cri-o://aba7d70f34fd8b10f906d1e3a247042a6979d08b086f44108d7d27fbe9a49c0c" gracePeriod=30
Oct 30 17:29:01.608644 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.608582    4497 generic.go:356] "Generic (PLEG): container finished" podID="fa4050d9-892d-47e9-adf4-96a72db5b1a5" containerID="aba7d70f34fd8b10f906d1e3a247042a6979d08b086f44108d7d27fbe9a49c0c" exitCode=0
Oct 30 17:29:01.608770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.608651    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs" event={"ID":"fa4050d9-892d-47e9-adf4-96a72db5b1a5","Type":"ContainerDied","Data":"aba7d70f34fd8b10f906d1e3a247042a6979d08b086f44108d7d27fbe9a49c0c"}
Oct 30 17:29:01.608770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.608701    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs" event={"ID":"fa4050d9-892d-47e9-adf4-96a72db5b1a5","Type":"ContainerDied","Data":"2574f3f7ef01f22b8beb52dbf2339a230e3f99bea91bcef13b7f14650cfe24a1"}
Oct 30 17:29:01.608770 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.608718    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2574f3f7ef01f22b8beb52dbf2339a230e3f99bea91bcef13b7f14650cfe24a1"
Oct 30 17:29:01.610485 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.610463    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 17:29:01.696971 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.696930    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-maas-api-tls\") pod \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") "
Oct 30 17:29:01.697172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.696992    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4050d9-892d-47e9-adf4-96a72db5b1a5-kube-api-access-bkfj5\" (UniqueName: \"kubernetes.io/projected/fa4050d9-892d-47e9-adf4-96a72db5b1a5-kube-api-access-bkfj5\") pod \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") "
Oct 30 17:29:01.697172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.697082    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-metrics-certs\") pod \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") "
Oct 30 17:29:01.697172 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.697125    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4050d9-892d-47e9-adf4-96a72db5b1a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4050d9-892d-47e9-adf4-96a72db5b1a5-openshift-service-ca\") pod \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\" (UID: \"fa4050d9-892d-47e9-adf4-96a72db5b1a5\") "
Oct 30 17:29:01.697508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.697485    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4050d9-892d-47e9-adf4-96a72db5b1a5-openshift-service-ca" pod "fa4050d9-892d-47e9-adf4-96a72db5b1a5" (UID: "fa4050d9-892d-47e9-adf4-96a72db5b1a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:01.699096 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.699068    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4050d9-892d-47e9-adf4-96a72db5b1a5-kube-api-access-bkfj5" pod "fa4050d9-892d-47e9-adf4-96a72db5b1a5" (UID: "fa4050d9-892d-47e9-adf4-96a72db5b1a5"). InnerVolumeSpecName "kube-api-access-bkfj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:01.699215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.699089    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-metrics-certs" pod "fa4050d9-892d-47e9-adf4-96a72db5b1a5" (UID: "fa4050d9-892d-47e9-adf4-96a72db5b1a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:01.699215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.699131    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-maas-api-tls" pod "fa4050d9-892d-47e9-adf4-96a72db5b1a5" (UID: "fa4050d9-892d-47e9-adf4-96a72db5b1a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:01.798419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.798388    4497 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4050d9-892d-47e9-adf4-96a72db5b1a5-openshift-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:01.798419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.798414    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-maas-api-tls\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:01.798419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.798425    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkfj5\" (UniqueName: \"kubernetes.io/projected/fa4050d9-892d-47e9-adf4-96a72db5b1a5-kube-api-access-bkfj5\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:01.798663 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:01.798433    4497 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa4050d9-892d-47e9-adf4-96a72db5b1a5-metrics-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:02.612839 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:02.612763    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"
Oct 30 17:29:02.655564 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:02.655527    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"]
Oct 30 17:29:02.665105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:02.665079    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ca339-68784f757b-dt7vs"]
Oct 30 17:29:04.061359 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.061306    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 17:29:04.122876 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.122851    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-plugins-config-volume\") pod \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\" (UID: \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\") "
Oct 30 17:29:04.123055 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.122902    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-kube-api-access-wfvkh\" (UniqueName: \"kubernetes.io/projected/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-kube-api-access-wfvkh\") pod \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\" (UID: \"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76\") "
Oct 30 17:29:04.123276 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.123252    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-plugins-config-volume" pod "b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" (UID: "b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:04.124921 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.124900    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-kube-api-access-wfvkh" pod "b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" (UID: "b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76"). InnerVolumeSpecName "kube-api-access-wfvkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:04.224159 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.224127    4497 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-plugins-config-volume\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:04.224159 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.224154    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvkh\" (UniqueName: \"kubernetes.io/projected/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76-kube-api-access-wfvkh\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:04.298721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.298681    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4050d9-892d-47e9-adf4-96a72db5b1a5" path="/var/lib/kubelet/pods/fa4050d9-892d-47e9-adf4-96a72db5b1a5/volumes"
Oct 30 17:29:04.623381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.623344    4497 generic.go:356] "Generic (PLEG): container finished" podID="b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" containerID="da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2" exitCode=0
Oct 30 17:29:04.623556 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.623402    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"
Oct 30 17:29:04.623556 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.623414    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt" event={"ID":"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76","Type":"ContainerDied","Data":"da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2"}
Oct 30 17:29:04.623556 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.623496    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt" event={"ID":"b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76","Type":"ContainerDied","Data":"4ec078a7bff46810f8056489ffe31d2307ed1e1613d28e61d810354a121395b2"}
Oct 30 17:29:04.623556 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.623524    4497 scope.go:122] "RemoveContainer" containerID="da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2"
Oct 30 17:29:04.632144 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.632117    4497 scope.go:122] "RemoveContainer" containerID="da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2"
Oct 30 17:29:04.632397 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:29:04.632376    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2\": container with ID starting with da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2 not found: ID does not exist" containerID="da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2"
Oct 30 17:29:04.632451 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.632405    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2"} err="failed to get container status \"da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2\": rpc error: code = NotFound desc = could not find container \"da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2\": container with ID starting with da2136fdb728d13e4fe67c4c9caa6d38d5e064fe15136f5d7424ef5dbfbd98a2 not found: ID does not exist"
Oct 30 17:29:04.668569 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.668541    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"]
Oct 30 17:29:04.677397 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:04.677372    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cf4621-5555fc486b-hbtnt"]
Oct 30 17:29:06.298018 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:06.297988    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" path="/var/lib/kubelet/pods/b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76/volumes"
Oct 30 17:29:10.232216 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232167    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"]
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232595    4497 cpu_manager.go:414] "containerMap: removing container" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232610    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232623    4497 cpu_manager.go:414] "containerMap: removing container" podUID="48c661b8-5a2a-4bb9-b12a-deb30e055e0b" containerName="maas-api"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232628    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c661b8-5a2a-4bb9-b12a-deb30e055e0b" containerName="maas-api"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232640    4497 cpu_manager.go:414] "containerMap: removing container" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232645    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232654    4497 cpu_manager.go:414] "containerMap: removing container" podUID="06ce2f6d-41de-4fbe-acd9-370f26fc69a3" containerName="maas-api"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232660    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ce2f6d-41de-4fbe-acd9-370f26fc69a3" containerName="maas-api"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232670    4497 cpu_manager.go:414] "containerMap: removing container" podUID="fa4050d9-892d-47e9-adf4-96a72db5b1a5" containerName="maas-api"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232678    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4050d9-892d-47e9-adf4-96a72db5b1a5" containerName="maas-api"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232692    4497 cpu_manager.go:414] "containerMap: removing container" podUID="b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" containerName="payload-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232697    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" containerName="payload-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232703    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" containerName="istio-proxy"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232708    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" containerName="istio-proxy"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232714    4497 cpu_manager.go:414] "containerMap: removing container" podUID="ed713416-d816-4724-bfd3-88ad575d63f2" containerName="payload-pre-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232719    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed713416-d816-4724-bfd3-88ad575d63f2" containerName="payload-pre-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232729    4497 cpu_manager.go:414] "containerMap: removing container" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232734    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232740    4497 cpu_manager.go:414] "containerMap: removing container" podUID="2592437d-ef7a-48d5-a319-35fe7287197d" containerName="payload-pre-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232745    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2592437d-ef7a-48d5-a319-35fe7287197d" containerName="payload-pre-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232757    4497 cpu_manager.go:414] "containerMap: removing container" podUID="c2eb9968-b523-4c4d-ae98-5183512a13ce" containerName="payload-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232763    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2eb9968-b523-4c4d-ae98-5183512a13ce" containerName="payload-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232774    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" containerName="payload-pre-processing"
Oct 30 17:29:10.232777 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232778    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" containerName="payload-pre-processing"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232808    4497 cpu_manager.go:414] "containerMap: removing container" podUID="918e1718-9153-44eb-8b8b-f26252675214" containerName="maas-api"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232813    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918e1718-9153-44eb-8b8b-f26252675214" containerName="maas-api"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232823    4497 cpu_manager.go:414] "containerMap: removing container" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232830    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232840    4497 cpu_manager.go:414] "containerMap: removing container" podUID="0e622cd5-6f48-4de4-af56-afd3a47279c5" containerName="payload-processing"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232845    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e622cd5-6f48-4de4-af56-afd3a47279c5" containerName="payload-processing"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232902    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232911    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="2592437d-ef7a-48d5-a319-35fe7287197d" containerName="payload-pre-processing"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232923    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2eb9968-b523-4c4d-ae98-5183512a13ce" containerName="payload-processing"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232930    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef79d9f-ddb1-46f4-a3bb-5376ed4ba300" containerName="payload-pre-processing"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232938    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0fca89-11f6-4450-9b7b-27f9fe014668" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232943    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4242887-cbd0-4d9a-ae41-732a0eec8669" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232948    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="918e1718-9153-44eb-8b8b-f26252675214" containerName="maas-api"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232952    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ee375e-34e5-45d1-b01c-61ce79ef4c33" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232958    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f019fe5f-39a5-4658-996c-4f91a78231e8" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232962    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="999975c1-3ece-490c-ad9f-a039410f320a" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232967    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e622cd5-6f48-4de4-af56-afd3a47279c5" containerName="payload-processing"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232972    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbff0ae7-4fa4-45d9-ad1d-fd8ec7b31690" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232978    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c661b8-5a2a-4bb9-b12a-deb30e055e0b" containerName="maas-api"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232985    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e7bdc0-1950-4c66-96d6-30caa8f8c2c0" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.232993    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ce2f6d-41de-4fbe-acd9-370f26fc69a3" containerName="maas-api"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.233001    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4050d9-892d-47e9-adf4-96a72db5b1a5" containerName="maas-api"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.233007    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a7a8c0-b7de-45c2-a3c7-289cf54f2a76" containerName="payload-processing"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.233012    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fac9f5-5b9c-47cf-9c53-710b21e23dac" containerName="istio-proxy"
Oct 30 17:29:10.233527 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.233017    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed713416-d816-4724-bfd3-88ad575d63f2" containerName="payload-pre-processing"
Oct 30 17:29:10.237295 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.237278    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.249223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.249207    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:29:10.249459 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.249444    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\""
Oct 30 17:29:10.259022 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.259000    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 17:29:10.259022 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.259007    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 17:29:10.261104 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.261085    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"]
Oct 30 17:29:10.280419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.280396    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.280551 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.280435    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.280648 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.280604    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdm4\" (UniqueName: \"kubernetes.io/projected/e987fae2-8d95-482c-88da-85ccc1ce56bd-kube-api-access-wkdm4\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.381754 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.381723    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdm4\" (UniqueName: \"kubernetes.io/projected/e987fae2-8d95-482c-88da-85ccc1ce56bd-kube-api-access-wkdm4\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.381907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.381797    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.381907 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.381841    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.382391 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.382370    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.384158 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.384137    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.403012 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.402991    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdm4\" (UniqueName: \"kubernetes.io/projected/e987fae2-8d95-482c-88da-85ccc1ce56bd-kube-api-access-wkdm4\") pod \"maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.549738 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.549707    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:10.848243 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:10.848158    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"]
Oct 30 17:29:10.853311 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:29:10.853283    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode987fae2_8d95_482c_88da_85ccc1ce56bd.slice/crio-0d87c8ad6c1fc24fbf45f00c6cdb072782c186df6c84818bfcacd8b93f3700d1 WatchSource:0}: Error finding container 0d87c8ad6c1fc24fbf45f00c6cdb072782c186df6c84818bfcacd8b93f3700d1: Status 404 returned error can't find the container with id 0d87c8ad6c1fc24fbf45f00c6cdb072782c186df6c84818bfcacd8b93f3700d1
Oct 30 17:29:11.659705 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:11.659671    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr" event={"ID":"e987fae2-8d95-482c-88da-85ccc1ce56bd","Type":"ContainerStarted","Data":"821a754dd8de214ce719e33ee649177aae198193e60da1c95ddeffca7cbaeeb5"}
Oct 30 17:29:11.660167 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:11.659725    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr" event={"ID":"e987fae2-8d95-482c-88da-85ccc1ce56bd","Type":"ContainerStarted","Data":"0d87c8ad6c1fc24fbf45f00c6cdb072782c186df6c84818bfcacd8b93f3700d1"}
Oct 30 17:29:11.695610 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:11.695565    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr" podStartSLOduration=1.695551708 podStartE2EDuration="1.695551708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:11.694557179 +0000 UTC m=+3981.548435442" watchObservedRunningTime="2026-10-30 17:29:11.695551708 +0000 UTC m=+3981.549429971"
Oct 30 17:29:12.666075 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:12.666036    4497 generic.go:356] "Generic (PLEG): container finished" podID="e987fae2-8d95-482c-88da-85ccc1ce56bd" containerID="821a754dd8de214ce719e33ee649177aae198193e60da1c95ddeffca7cbaeeb5" exitCode=0
Oct 30 17:29:12.666578 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:12.666117    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr" event={"ID":"e987fae2-8d95-482c-88da-85ccc1ce56bd","Type":"ContainerDied","Data":"821a754dd8de214ce719e33ee649177aae198193e60da1c95ddeffca7cbaeeb5"}
Oct 30 17:29:13.804633 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:13.804608    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:13.914745 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:13.914707    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e987fae2-8d95-482c-88da-85ccc1ce56bd-kube-api-access-wkdm4\" (UniqueName: \"kubernetes.io/projected/e987fae2-8d95-482c-88da-85ccc1ce56bd-kube-api-access-wkdm4\") pod \"e987fae2-8d95-482c-88da-85ccc1ce56bd\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") "
Oct 30 17:29:13.914935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:13.914766    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-openshift-ca\") pod \"e987fae2-8d95-482c-88da-85ccc1ce56bd\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") "
Oct 30 17:29:13.914935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:13.914814    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-xks-ca\") pod \"e987fae2-8d95-482c-88da-85ccc1ce56bd\" (UID: \"e987fae2-8d95-482c-88da-85ccc1ce56bd\") "
Oct 30 17:29:13.915219 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:13.915154    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-openshift-ca" pod "e987fae2-8d95-482c-88da-85ccc1ce56bd" (UID: "e987fae2-8d95-482c-88da-85ccc1ce56bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:13.916774 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:13.916755    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e987fae2-8d95-482c-88da-85ccc1ce56bd-kube-api-access-wkdm4" pod "e987fae2-8d95-482c-88da-85ccc1ce56bd" (UID: "e987fae2-8d95-482c-88da-85ccc1ce56bd"). InnerVolumeSpecName "kube-api-access-wkdm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:13.916898 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:13.916884    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-xks-ca" pod "e987fae2-8d95-482c-88da-85ccc1ce56bd" (UID: "e987fae2-8d95-482c-88da-85ccc1ce56bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:14.015843 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:14.015810    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-openshift-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:14.015843 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:14.015838    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e987fae2-8d95-482c-88da-85ccc1ce56bd-maas-api-xks-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:14.015843 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:14.015848    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdm4\" (UniqueName: \"kubernetes.io/projected/e987fae2-8d95-482c-88da-85ccc1ce56bd-kube-api-access-wkdm4\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:14.677160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:14.677082    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"
Oct 30 17:29:14.677160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:14.677099    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr" event={"ID":"e987fae2-8d95-482c-88da-85ccc1ce56bd","Type":"ContainerDied","Data":"0d87c8ad6c1fc24fbf45f00c6cdb072782c186df6c84818bfcacd8b93f3700d1"}
Oct 30 17:29:14.677160 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:14.677143    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d87c8ad6c1fc24fbf45f00c6cdb072782c186df6c84818bfcacd8b93f3700d1"
Oct 30 17:29:14.949726 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:14.949639    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"]
Oct 30 17:29:21.523037 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.523005    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"]
Oct 30 17:29:21.523590 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.523389    4497 cpu_manager.go:414] "containerMap: removing container" podUID="e987fae2-8d95-482c-88da-85ccc1ce56bd" containerName="revoke-keys"
Oct 30 17:29:21.523590 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.523403    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e987fae2-8d95-482c-88da-85ccc1ce56bd" containerName="revoke-keys"
Oct 30 17:29:21.523590 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.523506    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="e987fae2-8d95-482c-88da-85ccc1ce56bd" containerName="revoke-keys"
Oct 30 17:29:21.523590 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.523518    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="639dc55b-adf9-4a59-afc0-29f509b002b7" containerName="istio-proxy"
Oct 30 17:29:21.526972 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.526952    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.539626 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.539605    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 17:29:21.548962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.548924    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 17:29:21.548962 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.548947    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 17:29:21.549115 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.548975    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\""
Oct 30 17:29:21.550841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.550819    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"]
Oct 30 17:29:21.690293 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.690253    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.690480 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.690314    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.690480 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.690424    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm4rb\" (UniqueName: \"kubernetes.io/projected/51fe06c8-56a9-41c0-97af-1e918827fe64-kube-api-access-tm4rb\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.791463 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.791367    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.791463 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.791423    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.791684 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.791502    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm4rb\" (UniqueName: \"kubernetes.io/projected/51fe06c8-56a9-41c0-97af-1e918827fe64-kube-api-access-tm4rb\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.792107 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.792086    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.793745 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.793722    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.811921 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.811887    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm4rb\" (UniqueName: \"kubernetes.io/projected/51fe06c8-56a9-41c0-97af-1e918827fe64-kube-api-access-tm4rb\") pod \"maas-api-revoke-sub-e1644fdb-vmqb7\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.839888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:21.839861    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:21.919766 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:29:21.919725    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:29:22.156810 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:22.156729    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"]
Oct 30 17:29:22.161935 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:29:22.161907    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fe06c8_56a9_41c0_97af_1e918827fe64.slice/crio-d2d9044a6e93db4eed78373f7023f9031b6c9cd8a10cfe25d1fb4d5aae3e6cbf WatchSource:0}: Error finding container d2d9044a6e93db4eed78373f7023f9031b6c9cd8a10cfe25d1fb4d5aae3e6cbf: Status 404 returned error can't find the container with id d2d9044a6e93db4eed78373f7023f9031b6c9cd8a10cfe25d1fb4d5aae3e6cbf
Oct 30 17:29:22.717966 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:22.717927    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7" event={"ID":"51fe06c8-56a9-41c0-97af-1e918827fe64","Type":"ContainerStarted","Data":"ce41400ba6d92b84fe5e868434f435110247173617fd92367bd83d6f68a30d94"}
Oct 30 17:29:22.718390 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:22.717985    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7" event={"ID":"51fe06c8-56a9-41c0-97af-1e918827fe64","Type":"ContainerStarted","Data":"d2d9044a6e93db4eed78373f7023f9031b6c9cd8a10cfe25d1fb4d5aae3e6cbf"}
Oct 30 17:29:22.753419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:22.753376    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7" podStartSLOduration=1.753363444 podStartE2EDuration="1.753363444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:22.752762162 +0000 UTC m=+3992.606640426" watchObservedRunningTime="2026-10-30 17:29:22.753363444 +0000 UTC m=+3992.607241707"
Oct 30 17:29:23.724788 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:23.724747    4497 generic.go:356] "Generic (PLEG): container finished" podID="51fe06c8-56a9-41c0-97af-1e918827fe64" containerID="ce41400ba6d92b84fe5e868434f435110247173617fd92367bd83d6f68a30d94" exitCode=0
Oct 30 17:29:23.725226 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:23.724786    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7" event={"ID":"51fe06c8-56a9-41c0-97af-1e918827fe64","Type":"ContainerDied","Data":"ce41400ba6d92b84fe5e868434f435110247173617fd92367bd83d6f68a30d94"}
Oct 30 17:29:24.859408 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:24.859380    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:25.023138 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.023110    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-openshift-ca\") pod \"51fe06c8-56a9-41c0-97af-1e918827fe64\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") "
Oct 30 17:29:25.023363 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.023180    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-xks-ca\") pod \"51fe06c8-56a9-41c0-97af-1e918827fe64\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") "
Oct 30 17:29:25.023363 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.023245    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fe06c8-56a9-41c0-97af-1e918827fe64-kube-api-access-tm4rb\" (UniqueName: \"kubernetes.io/projected/51fe06c8-56a9-41c0-97af-1e918827fe64-kube-api-access-tm4rb\") pod \"51fe06c8-56a9-41c0-97af-1e918827fe64\" (UID: \"51fe06c8-56a9-41c0-97af-1e918827fe64\") "
Oct 30 17:29:25.023502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.023476    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-openshift-ca" pod "51fe06c8-56a9-41c0-97af-1e918827fe64" (UID: "51fe06c8-56a9-41c0-97af-1e918827fe64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:25.025286 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.025264    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-xks-ca" pod "51fe06c8-56a9-41c0-97af-1e918827fe64" (UID: "51fe06c8-56a9-41c0-97af-1e918827fe64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:25.025447 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.025428    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fe06c8-56a9-41c0-97af-1e918827fe64-kube-api-access-tm4rb" pod "51fe06c8-56a9-41c0-97af-1e918827fe64" (UID: "51fe06c8-56a9-41c0-97af-1e918827fe64"). InnerVolumeSpecName "kube-api-access-tm4rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:25.124131 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.124098    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-openshift-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:25.124131 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.124124    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fe06c8-56a9-41c0-97af-1e918827fe64-maas-api-xks-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:25.124131 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.124133    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm4rb\" (UniqueName: \"kubernetes.io/projected/51fe06c8-56a9-41c0-97af-1e918827fe64-kube-api-access-tm4rb\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:25.735181 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.735155    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7"
Oct 30 17:29:25.735346 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.735154    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1644fdb-vmqb7" event={"ID":"51fe06c8-56a9-41c0-97af-1e918827fe64","Type":"ContainerDied","Data":"d2d9044a6e93db4eed78373f7023f9031b6c9cd8a10cfe25d1fb4d5aae3e6cbf"}
Oct 30 17:29:25.735346 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:25.735305    4497 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d9044a6e93db4eed78373f7023f9031b6c9cd8a10cfe25d1fb4d5aae3e6cbf"
Oct 30 17:29:29.969786 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:29.969719    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main" containerID="cri-o://36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5" gracePeriod=45
Oct 30 17:29:30.212651 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.212625    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 17:29:30.376760 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.376725    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tmp-dir\") pod \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") "
Oct 30 17:29:30.376760 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.376762    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-home\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-home\") pod \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") "
Oct 30 17:29:30.376999 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.376797    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-dshm\") pod \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") "
Oct 30 17:29:30.376999 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.376956    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-kube-api-access-79tgd\" (UniqueName: \"kubernetes.io/projected/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-kube-api-access-79tgd\") pod \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") "
Oct 30 17:29:30.377093 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.377002    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tls-certs\") pod \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") "
Oct 30 17:29:30.377093 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.377028    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-model-cache\") pod \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\" (UID: \"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f\") "
Oct 30 17:29:30.377093 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.377082    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-home" pod "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" (UID: "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:30.377226 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.377102    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tmp-dir" pod "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" (UID: "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:30.377474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.377451    4497 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tmp-dir\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:30.377474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.377449    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-model-cache" pod "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" (UID: "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:30.377678 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.377476    4497 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-home\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:30.379520 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.379483    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-kube-api-access-79tgd" pod "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" (UID: "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f"). InnerVolumeSpecName "kube-api-access-79tgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:30.379640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.379544    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-dshm" pod "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" (UID: "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:30.379640 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.379559    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tls-certs" pod "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" (UID: "7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 17:29:30.478666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.478630    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79tgd\" (UniqueName: \"kubernetes.io/projected/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-kube-api-access-79tgd\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:30.478666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.478661    4497 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-tls-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:30.478666 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.478671    4497 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-model-cache\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:30.478903 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.478679    4497 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f-dshm\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:30.760896 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.760865    4497 generic.go:356] "Generic (PLEG): container finished" podID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerID="36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5" exitCode=0
Oct 30 17:29:30.761057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.760909    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"
Oct 30 17:29:30.761057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.760948    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" event={"ID":"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f","Type":"ContainerDied","Data":"36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5"}
Oct 30 17:29:30.761057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.760994    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn" event={"ID":"7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f","Type":"ContainerDied","Data":"3c9682c059aba300d8c9e0f8a61eb86b8643bb61159b6b1f35530f227c70bc42"}
Oct 30 17:29:30.761057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.761015    4497 scope.go:122] "RemoveContainer" containerID="36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5"
Oct 30 17:29:30.769860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.769844    4497 scope.go:122] "RemoveContainer" containerID="36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5"
Oct 30 17:29:30.770111 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:29:30.770093    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5\": container with ID starting with 36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5 not found: ID does not exist" containerID="36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5"
Oct 30 17:29:30.770156 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.770120    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5"} err="failed to get container status \"36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5\": rpc error: code = NotFound desc = could not find container \"36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5\": container with ID starting with 36231a20865c0c2d1353ab7374593247f5758fb3c32a00dc7a73ea1da03df6f5 not found: ID does not exist"
Oct 30 17:29:30.804419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.804366    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"]
Oct 30 17:29:30.813860 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:30.813836    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-2-w1-a77b48-kserve-744cc47bbb-q9dtn"]
Oct 30 17:29:32.298020 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:32.297946    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" path="/var/lib/kubelet/pods/7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f/volumes"
Oct 30 17:29:37.378424 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:37.378392    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"]
Oct 30 17:29:37.387655 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:37.387631    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a77b48-xhbcr"]
Oct 30 17:29:38.298169 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:38.298132    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e987fae2-8d95-482c-88da-85ccc1ce56bd" path="/var/lib/kubelet/pods/e987fae2-8d95-482c-88da-85ccc1ce56bd/volumes"
Oct 30 17:29:44.398046 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:44.398011    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"]
Oct 30 17:29:44.398561 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:44.398279    4497 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy" containerID="cri-o://cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544" gracePeriod=30
Oct 30 17:29:46.291147 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:46.291121    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-86rqx_aea41ce5-ec7f-4bb6-a64f-458422b8566d/kuadrant-console-plugin/0.log"
Oct 30 17:29:49.613118 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.613086    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gbc6r_b467741e-e532-40ad-9261-db4f8b0cac24/server/0.log"
Oct 30 17:29:49.648102 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.648072    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 17:29:49.749769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749694    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-certs\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.749769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749738    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-podinfo\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.749769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749757    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-data\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.750057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749786    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-socket\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.750057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749825    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-token\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-token\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.750057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749878    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-kube-api-access-4r6j9\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-kube-api-access-4r6j9\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.750057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749924    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istiod-ca-cert\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.750057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749980    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-credential-socket\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.750057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.749986    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-certs" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:49.750057 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750028    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-envoy\") pod \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\" (UID: \"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c\") "
Oct 30 17:29:49.750431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750146    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-socket" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:49.750431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750166    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-data" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:49.750431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750362    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istiod-ca-cert" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:29:49.750431 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750359    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-credential-socket" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:49.750583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750497    4497 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istiod-ca-cert\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.750583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750518    4497 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-credential-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.750583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750533    4497 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-certs\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.750583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750547    4497 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-data\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.750583 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.750560    4497 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-workload-socket\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.752105 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.752085    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-envoy" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 17:29:49.752233 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.752215    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-token" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:49.752476 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.752446    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-kube-api-access-4r6j9" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "kube-api-access-4r6j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:29:49.752476 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.752463    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-podinfo" pod "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" (UID: "9b17bcb4-d397-4a7f-a067-a5fbf0265b1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 17:29:49.855139 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.855107    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r6j9\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-kube-api-access-4r6j9\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.855348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.855136    4497 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-envoy\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.855348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.855273    4497 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-podinfo\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.855348 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.855283    4497 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c-istio-token\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:29:49.856166 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.856141    4497 generic.go:356] "Generic (PLEG): container finished" podID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerID="cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544" exitCode=0
Oct 30 17:29:49.856268 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.856219    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"
Oct 30 17:29:49.856268 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.856221    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" event={"ID":"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c","Type":"ContainerDied","Data":"cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544"}
Oct 30 17:29:49.856345 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.856306    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh" event={"ID":"9b17bcb4-d397-4a7f-a067-a5fbf0265b1c","Type":"ContainerDied","Data":"44e22a4b094b63b5a1077fc39d51054f53836d400b89c343c6a286343179abc3"}
Oct 30 17:29:49.856345 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.856329    4497 scope.go:122] "RemoveContainer" containerID="cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544"
Oct 30 17:29:49.865376 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.865355    4497 scope.go:122] "RemoveContainer" containerID="cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544"
Oct 30 17:29:49.865620 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:29:49.865604    4497 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544\": container with ID starting with cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544 not found: ID does not exist" containerID="cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544"
Oct 30 17:29:49.865660 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.865627    4497 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544"} err="failed to get container status \"cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544\": rpc error: code = NotFound desc = could not find container \"cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544\": container with ID starting with cd04da49fec8f2c5309c9c008bee53fb6debf0255a56ee2addd6a099ab247544 not found: ID does not exist"
Oct 30 17:29:49.899638 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.899605    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"]
Oct 30 17:29:49.905596 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:49.905575    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-sshnh"]
Oct 30 17:29:50.298280 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:50.298240    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" path="/var/lib/kubelet/pods/9b17bcb4-d397-4a7f-a067-a5fbf0265b1c/volumes"
Oct 30 17:29:51.915000 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:29:51.914966    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:29:52.312030 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:29:52.311999    4497 scope.go:122] "RemoveContainer" containerID="aba7d70f34fd8b10f906d1e3a247042a6979d08b086f44108d7d27fbe9a49c0c"
Oct 30 17:30:00.160516 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160478    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"]
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160826    4497 cpu_manager.go:414] "containerMap: removing container" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main"
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160836    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main"
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160852    4497 cpu_manager.go:414] "containerMap: removing container" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy"
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160857    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy"
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160886    4497 cpu_manager.go:414] "containerMap: removing container" podUID="51fe06c8-56a9-41c0-97af-1e918827fe64" containerName="revoke-keys"
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160893    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fe06c8-56a9-41c0-97af-1e918827fe64" containerName="revoke-keys"
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160953    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c003d2c-031f-4b5e-a6dc-4c88bd8fa03f" containerName="main"
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160963    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b17bcb4-d397-4a7f-a067-a5fbf0265b1c" containerName="istio-proxy"
Oct 30 17:30:00.161087 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.160975    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fe06c8-56a9-41c0-97af-1e918827fe64" containerName="revoke-keys"
Oct 30 17:30:00.164031 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.164011    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:00.185985 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.185957    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\""
Oct 30 17:30:00.186166 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.185978    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 17:30:00.186279 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.186007    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 17:30:00.188171 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.188147    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"]
Oct 30 17:30:00.252065 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.252031    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zvv\" (UniqueName: \"kubernetes.io/projected/f0674aeb-462f-4ef4-922d-46206ada151a-kube-api-access-f2zvv\") pod \"maas-api-key-cleanup-29889690-lqswl\" (UID: \"f0674aeb-462f-4ef4-922d-46206ada151a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:00.252259 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.252087    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0674aeb-462f-4ef4-922d-46206ada151a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-lqswl\" (UID: \"f0674aeb-462f-4ef4-922d-46206ada151a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:00.353139 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.353099    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zvv\" (UniqueName: \"kubernetes.io/projected/f0674aeb-462f-4ef4-922d-46206ada151a-kube-api-access-f2zvv\") pod \"maas-api-key-cleanup-29889690-lqswl\" (UID: \"f0674aeb-462f-4ef4-922d-46206ada151a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:00.353139 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.353140    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0674aeb-462f-4ef4-922d-46206ada151a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-lqswl\" (UID: \"f0674aeb-462f-4ef4-922d-46206ada151a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:00.353782 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.353761    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0674aeb-462f-4ef4-922d-46206ada151a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889690-lqswl\" (UID: \"f0674aeb-462f-4ef4-922d-46206ada151a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:00.375313 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.375281    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zvv\" (UniqueName: \"kubernetes.io/projected/f0674aeb-462f-4ef4-922d-46206ada151a-kube-api-access-f2zvv\") pod \"maas-api-key-cleanup-29889690-lqswl\" (UID: \"f0674aeb-462f-4ef4-922d-46206ada151a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:00.476877 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.476796    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:00.772495 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.772464    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"]
Oct 30 17:30:00.777466 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:30:00.777439    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0674aeb_462f_4ef4_922d_46206ada151a.slice/crio-ddfe8248650264653f5982192f19165307ae2a256da186d26bf47c4e7289424e WatchSource:0}: Error finding container ddfe8248650264653f5982192f19165307ae2a256da186d26bf47c4e7289424e: Status 404 returned error can't find the container with id ddfe8248650264653f5982192f19165307ae2a256da186d26bf47c4e7289424e
Oct 30 17:30:00.905712 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:00.905673    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" event={"ID":"f0674aeb-462f-4ef4-922d-46206ada151a","Type":"ContainerStarted","Data":"ddfe8248650264653f5982192f19165307ae2a256da186d26bf47c4e7289424e"}
Oct 30 17:30:01.911319 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:01.911233    4497 generic.go:356] "Generic (PLEG): container finished" podID="f0674aeb-462f-4ef4-922d-46206ada151a" containerID="3a43a511f46c37956e523479679c287ef6f5c6f3cda6bf4bf040d87026045314" exitCode=60
Oct 30 17:30:01.911319 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:01.911309    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" event={"ID":"f0674aeb-462f-4ef4-922d-46206ada151a","Type":"ContainerDied","Data":"3a43a511f46c37956e523479679c287ef6f5c6f3cda6bf4bf040d87026045314"}
Oct 30 17:30:01.911727 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:01.911516    4497 scope.go:122] "RemoveContainer" containerID="3a43a511f46c37956e523479679c287ef6f5c6f3cda6bf4bf040d87026045314"
Oct 30 17:30:02.781721 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:02.781696    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gbc6r_b467741e-e532-40ad-9261-db4f8b0cac24/server/0.log"
Oct 30 17:30:02.926298 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:02.926210    4497 generic.go:356] "Generic (PLEG): container finished" podID="f0674aeb-462f-4ef4-922d-46206ada151a" containerID="617f6502dce22adf425a1dd049ba6debfc9410b8d22e8b9551518626b5d0f3ab" exitCode=60
Oct 30 17:30:02.926298 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:02.926263    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" event={"ID":"f0674aeb-462f-4ef4-922d-46206ada151a","Type":"ContainerDied","Data":"617f6502dce22adf425a1dd049ba6debfc9410b8d22e8b9551518626b5d0f3ab"}
Oct 30 17:30:02.926815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:02.926346    4497 scope.go:122] "RemoveContainer" containerID="3a43a511f46c37956e523479679c287ef6f5c6f3cda6bf4bf040d87026045314"
Oct 30 17:30:02.926815 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:02.926600    4497 scope.go:122] "RemoveContainer" containerID="617f6502dce22adf425a1dd049ba6debfc9410b8d22e8b9551518626b5d0f3ab"
Oct 30 17:30:02.926924 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:30:02.926863    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-lqswl_odh-ai-gateway-infra(f0674aeb-462f-4ef4-922d-46206ada151a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" podUID="f0674aeb-462f-4ef4-922d-46206ada151a"
Oct 30 17:30:03.932630 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:03.932603    4497 scope.go:122] "RemoveContainer" containerID="617f6502dce22adf425a1dd049ba6debfc9410b8d22e8b9551518626b5d0f3ab"
Oct 30 17:30:03.933091 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:30:03.932876    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-lqswl_odh-ai-gateway-infra(f0674aeb-462f-4ef4-922d-46206ada151a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" podUID="f0674aeb-462f-4ef4-922d-46206ada151a"
Oct 30 17:30:04.143624 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:04.143595    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29889690-lqswl_f0674aeb-462f-4ef4-922d-46206ada151a/cleanup/1.log"
Oct 30 17:30:05.864045 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:05.863960    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e1644fdb-vmqb7_51fe06c8-56a9-41c0-97af-1e918827fe64/revoke-keys/0.log"
Oct 30 17:30:06.665200 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:06.665168    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29889690-lqswl_f0674aeb-462f-4ef4-922d-46206ada151a/cleanup/1.log"
Oct 30 17:30:08.565149 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:08.565120    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-86rqx_aea41ce5-ec7f-4bb6-a64f-458422b8566d/kuadrant-console-plugin/0.log"
Oct 30 17:30:10.682275 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:10.682248    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-bz58x_941eec69-d000-41b3-9250-3ce425d3f9e7/kube-auth-proxy/0.log"
Oct 30 17:30:11.062150 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:11.062121    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-jzt7r_c92b23e0-1211-499e-a41d-ab0ae8387659/istio-proxy/0.log"
Oct 30 17:30:12.974176 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:12.974143    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-fnwwb_97f4feda-8781-4d4f-9dd5-77f0a5384990/main/0.log"
Oct 30 17:30:13.178830 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:13.178794    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-v4pm6_55a8a020-fd10-4aa9-9b1b-12c100212c87/main/0.log"
Oct 30 17:30:13.739345 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:13.739284    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zxxxq_64cadc03-d87e-4a64-aa44-8e97d58c0dc5/main/0.log"
Oct 30 17:30:14.129320 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:14.129291    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zw75x_5e27b12c-daa9-49eb-9f5a-0815ec044446/main/0.log"
Oct 30 17:30:16.292542 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:16.292511    4497 scope.go:122] "RemoveContainer" containerID="617f6502dce22adf425a1dd049ba6debfc9410b8d22e8b9551518626b5d0f3ab"
Oct 30 17:30:16.997751 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:16.997665    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" event={"ID":"f0674aeb-462f-4ef4-922d-46206ada151a","Type":"ContainerStarted","Data":"389a20c24224d79afdce0755e9e96a83cc274c2ac775047c592eea2830549e65"}
Oct 30 17:30:17.034807 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:17.034761    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" podStartSLOduration=17.034747888 podStartE2EDuration="17.034747888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:17.032408582 +0000 UTC m=+4046.886286845" watchObservedRunningTime="2026-10-30 17:30:17.034747888 +0000 UTC m=+4046.888626151"
Oct 30 17:30:18.003407 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:18.003370    4497 generic.go:356] "Generic (PLEG): container finished" podID="f0674aeb-462f-4ef4-922d-46206ada151a" containerID="389a20c24224d79afdce0755e9e96a83cc274c2ac775047c592eea2830549e65" exitCode=60
Oct 30 17:30:18.003904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:18.003443    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" event={"ID":"f0674aeb-462f-4ef4-922d-46206ada151a","Type":"ContainerDied","Data":"389a20c24224d79afdce0755e9e96a83cc274c2ac775047c592eea2830549e65"}
Oct 30 17:30:18.003904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:18.003504    4497 scope.go:122] "RemoveContainer" containerID="617f6502dce22adf425a1dd049ba6debfc9410b8d22e8b9551518626b5d0f3ab"
Oct 30 17:30:18.003904 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:18.003861    4497 scope.go:122] "RemoveContainer" containerID="389a20c24224d79afdce0755e9e96a83cc274c2ac775047c592eea2830549e65"
Oct 30 17:30:18.004142 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:30:18.004120    4497 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889690-lqswl_odh-ai-gateway-infra(f0674aeb-462f-4ef4-922d-46206ada151a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" podUID="f0674aeb-462f-4ef4-922d-46206ada151a"
Oct 30 17:30:18.050366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:18.050262    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"]
Oct 30 17:30:19.140417 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:19.140383    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:19.225324 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:19.225288    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0674aeb-462f-4ef4-922d-46206ada151a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0674aeb-462f-4ef4-922d-46206ada151a-maas-api-service-ca\") pod \"f0674aeb-462f-4ef4-922d-46206ada151a\" (UID: \"f0674aeb-462f-4ef4-922d-46206ada151a\") "
Oct 30 17:30:19.225502 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:19.225365    4497 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0674aeb-462f-4ef4-922d-46206ada151a-kube-api-access-f2zvv\" (UniqueName: \"kubernetes.io/projected/f0674aeb-462f-4ef4-922d-46206ada151a-kube-api-access-f2zvv\") pod \"f0674aeb-462f-4ef4-922d-46206ada151a\" (UID: \"f0674aeb-462f-4ef4-922d-46206ada151a\") "
Oct 30 17:30:19.225641 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:19.225615    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0674aeb-462f-4ef4-922d-46206ada151a-maas-api-service-ca" pod "f0674aeb-462f-4ef4-922d-46206ada151a" (UID: "f0674aeb-462f-4ef4-922d-46206ada151a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 17:30:19.228215 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:19.228176    4497 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0674aeb-462f-4ef4-922d-46206ada151a-kube-api-access-f2zvv" pod "f0674aeb-462f-4ef4-922d-46206ada151a" (UID: "f0674aeb-462f-4ef4-922d-46206ada151a"). InnerVolumeSpecName "kube-api-access-f2zvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 17:30:19.326736 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:19.326642    4497 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0674aeb-462f-4ef4-922d-46206ada151a-maas-api-service-ca\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:30:19.326736 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:19.326676    4497 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zvv\" (UniqueName: \"kubernetes.io/projected/f0674aeb-462f-4ef4-922d-46206ada151a-kube-api-access-f2zvv\") on node \"ip-10-0-136-74.ec2.internal\" DevicePath \"\""
Oct 30 17:30:20.018017 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:20.017984    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl" event={"ID":"f0674aeb-462f-4ef4-922d-46206ada151a","Type":"ContainerDied","Data":"ddfe8248650264653f5982192f19165307ae2a256da186d26bf47c4e7289424e"}
Oct 30 17:30:20.018224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:20.018045    4497 scope.go:122] "RemoveContainer" containerID="389a20c24224d79afdce0755e9e96a83cc274c2ac775047c592eea2830549e65"
Oct 30 17:30:20.018224 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:20.017998    4497 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"
Oct 30 17:30:20.066511 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:20.066484    4497 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"]
Oct 30 17:30:20.075912 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:20.075890    4497 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889690-lqswl"]
Oct 30 17:30:20.299543 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:20.299464    4497 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" path="/var/lib/kubelet/pods/f0674aeb-462f-4ef4-922d-46206ada151a/volumes"
Oct 30 17:30:21.915085 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:30:21.915051    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:30:24.497997 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.497958    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vt4kv/must-gather-vbl5t"]
Oct 30 17:30:24.499463 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.499370    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.499635 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.499621    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.499776 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.499765    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.499941 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.499927    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.500395 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.500375    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.500545 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.500535    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.500639 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.500630    4497 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.500940 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.500927    4497 cpu_manager.go:414] "containerMap: removing container" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.501032 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.501021    4497 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0674aeb-462f-4ef4-922d-46206ada151a" containerName="cleanup"
Oct 30 17:30:24.506797 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.506774    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vt4kv/must-gather-vbl5t"
Oct 30 17:30:24.521306 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.521280    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vt4kv/must-gather-vbl5t"]
Oct 30 17:30:24.529758 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.529738    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vt4kv\"/\"default-dockercfg-k28r8\""
Oct 30 17:30:24.529758 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.529756    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vt4kv\"/\"openshift-service-ca.crt\""
Oct 30 17:30:24.529888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.529809    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vt4kv\"/\"kube-root-ca.crt\""
Oct 30 17:30:24.570888 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.570862    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8gd\" (UniqueName: \"kubernetes.io/projected/d1338e0d-6d34-48b2-8816-daf2a36be768-kube-api-access-gd8gd\") pod \"must-gather-vbl5t\" (UID: \"d1338e0d-6d34-48b2-8816-daf2a36be768\") " pod="openshift-must-gather-vt4kv/must-gather-vbl5t"
Oct 30 17:30:24.571028 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.570925    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d1338e0d-6d34-48b2-8816-daf2a36be768-must-gather-output\") pod \"must-gather-vbl5t\" (UID: \"d1338e0d-6d34-48b2-8816-daf2a36be768\") " pod="openshift-must-gather-vt4kv/must-gather-vbl5t"
Oct 30 17:30:24.671474 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.671441    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8gd\" (UniqueName: \"kubernetes.io/projected/d1338e0d-6d34-48b2-8816-daf2a36be768-kube-api-access-gd8gd\") pod \"must-gather-vbl5t\" (UID: \"d1338e0d-6d34-48b2-8816-daf2a36be768\") " pod="openshift-must-gather-vt4kv/must-gather-vbl5t"
Oct 30 17:30:24.671652 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.671513    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d1338e0d-6d34-48b2-8816-daf2a36be768-must-gather-output\") pod \"must-gather-vbl5t\" (UID: \"d1338e0d-6d34-48b2-8816-daf2a36be768\") " pod="openshift-must-gather-vt4kv/must-gather-vbl5t"
Oct 30 17:30:24.671823 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.671807    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d1338e0d-6d34-48b2-8816-daf2a36be768-must-gather-output\") pod \"must-gather-vbl5t\" (UID: \"d1338e0d-6d34-48b2-8816-daf2a36be768\") " pod="openshift-must-gather-vt4kv/must-gather-vbl5t"
Oct 30 17:30:24.692251 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.692220    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8gd\" (UniqueName: \"kubernetes.io/projected/d1338e0d-6d34-48b2-8816-daf2a36be768-kube-api-access-gd8gd\") pod \"must-gather-vbl5t\" (UID: \"d1338e0d-6d34-48b2-8816-daf2a36be768\") " pod="openshift-must-gather-vt4kv/must-gather-vbl5t"
Oct 30 17:30:24.820121 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:24.820089    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vt4kv/must-gather-vbl5t"
Oct 30 17:30:25.114466 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:25.114386    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vt4kv/must-gather-vbl5t"]
Oct 30 17:30:25.119129 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:30:25.119102    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1338e0d_6d34_48b2_8816_daf2a36be768.slice/crio-2b1d54be93024a882cd9977ee08bd6f42ff730be0429fddd3b92a87cd909d5aa WatchSource:0}: Error finding container 2b1d54be93024a882cd9977ee08bd6f42ff730be0429fddd3b92a87cd909d5aa: Status 404 returned error can't find the container with id 2b1d54be93024a882cd9977ee08bd6f42ff730be0429fddd3b92a87cd909d5aa
Oct 30 17:30:25.120772 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:25.120752    4497 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 17:30:26.048386 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:26.048349    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vt4kv/must-gather-vbl5t" event={"ID":"d1338e0d-6d34-48b2-8816-daf2a36be768","Type":"ContainerStarted","Data":"2b1d54be93024a882cd9977ee08bd6f42ff730be0429fddd3b92a87cd909d5aa"}
Oct 30 17:30:29.458247 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:29.458215    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x4pp5_536629a7-e02f-4d49-a9de-cccd0e39132e/global-pull-secret-syncer/0.log"
Oct 30 17:30:30.281209 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:30.281161    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j6fh2_3c365930-eba1-4476-8a6c-7efe851313a8/konnectivity-agent/0.log"
Oct 30 17:30:30.871577 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:30.871503    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-74.ec2.internal_041057424ec306b79ef7363e62fcbac0/haproxy/0.log"
Oct 30 17:30:31.078982 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:31.078940    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vt4kv/must-gather-vbl5t" event={"ID":"d1338e0d-6d34-48b2-8816-daf2a36be768","Type":"ContainerStarted","Data":"10e8cf4deea1e4a9ac47686e0f3777245cd547dab943532aeb5b1888b27c006f"}
Oct 30 17:30:31.079169 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:31.078997    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vt4kv/must-gather-vbl5t" event={"ID":"d1338e0d-6d34-48b2-8816-daf2a36be768","Type":"ContainerStarted","Data":"ba0089679799a59edd89c90255919c4a6a2f73dc314378ec76669ac3d1dbe7ac"}
Oct 30 17:30:31.119989 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:31.119935    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vt4kv/must-gather-vbl5t" podStartSLOduration=1.872362183 podStartE2EDuration="7.119920616s" totalImagesPullingTime="5.247558433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:31.117852677 +0000 UTC m=+4060.971730942" watchObservedRunningTime="2026-10-30 17:30:31.119920616 +0000 UTC m=+4060.973798888"
Oct 30 17:30:35.076403 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:35.076365    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gbc6r_b467741e-e532-40ad-9261-db4f8b0cac24/server/0.log"
Oct 30 17:30:35.257820 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:35.257785    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gbc6r_b467741e-e532-40ad-9261-db4f8b0cac24/server/0.log"
Oct 30 17:30:39.448921 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.448883    4497 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"]
Oct 30 17:30:39.457258 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.457236    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.479672 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.479637    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skgpd\"/\"openshift-service-ca.crt\""
Oct 30 17:30:39.479672 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.479645    4497 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-skgpd\"/\"default-dockercfg-fv925\""
Oct 30 17:30:39.479841 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.479671    4497 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skgpd\"/\"kube-root-ca.crt\""
Oct 30 17:30:39.481297 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.481279    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"]
Oct 30 17:30:39.520694 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.520664    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-sys\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.520835 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.520705    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-lib-modules\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.520835 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.520825    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-proc\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.520940 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.520857    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n466\" (UniqueName: \"kubernetes.io/projected/b1d88411-218b-4e9d-a724-caab8fd75aeb-kube-api-access-6n466\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.520940 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.520887    4497 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-podres\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622171 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622139    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-proc\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622197    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n466\" (UniqueName: \"kubernetes.io/projected/b1d88411-218b-4e9d-a724-caab8fd75aeb-kube-api-access-6n466\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622294    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-proc\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622381 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622326    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-podres\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622530 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622380    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-sys\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622530 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622416    4497 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-lib-modules\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622530 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622440    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-podres\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622530 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622482    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-sys\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.622678 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.622544    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1d88411-218b-4e9d-a724-caab8fd75aeb-lib-modules\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.645758 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.645719    4497 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n466\" (UniqueName: \"kubernetes.io/projected/b1d88411-218b-4e9d-a724-caab8fd75aeb-kube-api-access-6n466\") pod \"perf-node-gather-daemonset-cv5kj\" (UID: \"b1d88411-218b-4e9d-a724-caab8fd75aeb\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.770872 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.770843    4497 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:39.834261 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:39.833232    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-86rqx_aea41ce5-ec7f-4bb6-a64f-458422b8566d/kuadrant-console-plugin/0.log"
Oct 30 17:30:40.071366 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:40.071275    4497 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"]
Oct 30 17:30:40.076120 ip-10-0-136-74 kubenswrapper[4497]: W1030 17:30:40.076095    4497 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb1d88411_218b_4e9d_a724_caab8fd75aeb.slice/crio-af81c4848d5f777d3f5b18dcd2055e7205f4a1d3db182c9746868236e51acc81 WatchSource:0}: Error finding container af81c4848d5f777d3f5b18dcd2055e7205f4a1d3db182c9746868236e51acc81: Status 404 returned error can't find the container with id af81c4848d5f777d3f5b18dcd2055e7205f4a1d3db182c9746868236e51acc81
Oct 30 17:30:40.126465 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:40.126433    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj" event={"ID":"b1d88411-218b-4e9d-a724-caab8fd75aeb","Type":"ContainerStarted","Data":"af81c4848d5f777d3f5b18dcd2055e7205f4a1d3db182c9746868236e51acc81"}
Oct 30 17:30:41.132645 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:41.132602    4497 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj" event={"ID":"b1d88411-218b-4e9d-a724-caab8fd75aeb","Type":"ContainerStarted","Data":"7be3364ba584fb73c7abf072173200e4c2e52f7d7ff4f3a38729d267a6391dce"}
Oct 30 17:30:41.133217 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:41.132706    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:41.169003 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:41.168957    4497 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj" podStartSLOduration=2.168944846 podStartE2EDuration="2.168944846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:41.167774721 +0000 UTC m=+4071.021652989" watchObservedRunningTime="2026-10-30 17:30:41.168944846 +0000 UTC m=+4071.022823111"
Oct 30 17:30:43.093234 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:43.093167    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gbc6r_b467741e-e532-40ad-9261-db4f8b0cac24/server/0.log"
Oct 30 17:30:43.190587 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:43.190561    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gbc6r_b467741e-e532-40ad-9261-db4f8b0cac24/server/0.log"
Oct 30 17:30:43.561493 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:43.561458    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67f88ddd8c-vpdnc_fddb9fb8-e302-4783-97b3-db16eec92029/metrics-server/0.log"
Oct 30 17:30:43.685473 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:43.685443    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-xjbdh_625a8fb0-a249-4439-ab60-04fa8d52b0b6/monitoring-plugin/0.log"
Oct 30 17:30:43.810508 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:43.810479    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bl4n4_007bee35-f9a2-4595-9409-624a42b6abfa/node-exporter/0.log"
Oct 30 17:30:43.929338 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:43.929261    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bl4n4_007bee35-f9a2-4595-9409-624a42b6abfa/kube-rbac-proxy/0.log"
Oct 30 17:30:44.048769 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:44.048714    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bl4n4_007bee35-f9a2-4595-9409-624a42b6abfa/init-textfile/0.log"
Oct 30 17:30:45.494088 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:45.494053    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_944d1bde-31c5-4c49-8d29-600c5b2feb1e/prometheus/0.log"
Oct 30 17:30:45.613625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:45.613592    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_944d1bde-31c5-4c49-8d29-600c5b2feb1e/config-reloader/0.log"
Oct 30 17:30:45.734298 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:45.734244    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_944d1bde-31c5-4c49-8d29-600c5b2feb1e/thanos-sidecar/0.log"
Oct 30 17:30:45.859136 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:45.859103    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_944d1bde-31c5-4c49-8d29-600c5b2feb1e/kube-rbac-proxy-web/0.log"
Oct 30 17:30:45.980781 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:45.980749    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_944d1bde-31c5-4c49-8d29-600c5b2feb1e/kube-rbac-proxy/0.log"
Oct 30 17:30:46.101361 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:46.101329    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_944d1bde-31c5-4c49-8d29-600c5b2feb1e/kube-rbac-proxy-thanos/0.log"
Oct 30 17:30:46.223625 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:46.223524    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_944d1bde-31c5-4c49-8d29-600c5b2feb1e/init-config-reloader/0.log"
Oct 30 17:30:46.352419 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:46.352388    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8vg7x_5f171456-2c42-41ef-b11a-352f6d543e37/prometheus-operator/0.log"
Oct 30 17:30:46.469364 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:46.469338    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8vg7x_5f171456-2c42-41ef-b11a-352f6d543e37/kube-rbac-proxy/0.log"
Oct 30 17:30:47.126128 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:47.126093    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb4d4fc79-vknk2_9ada3a96-8c0d-4ec0-b3f4-24821f397f3c/thanos-query/0.log"
Oct 30 17:30:47.147623 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:47.147593    4497 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-cv5kj"
Oct 30 17:30:47.246894 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:47.246862    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb4d4fc79-vknk2_9ada3a96-8c0d-4ec0-b3f4-24821f397f3c/kube-rbac-proxy-web/0.log"
Oct 30 17:30:47.368285 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:47.368247    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb4d4fc79-vknk2_9ada3a96-8c0d-4ec0-b3f4-24821f397f3c/kube-rbac-proxy/0.log"
Oct 30 17:30:47.488049 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:47.487974    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb4d4fc79-vknk2_9ada3a96-8c0d-4ec0-b3f4-24821f397f3c/prom-label-proxy/0.log"
Oct 30 17:30:47.609139 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:47.609096    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb4d4fc79-vknk2_9ada3a96-8c0d-4ec0-b3f4-24821f397f3c/kube-rbac-proxy-rules/0.log"
Oct 30 17:30:47.730369 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:47.730336    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb4d4fc79-vknk2_9ada3a96-8c0d-4ec0-b3f4-24821f397f3c/kube-rbac-proxy-metrics/0.log"
Oct 30 17:30:50.822765 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:50.822736    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gbc6r_b467741e-e532-40ad-9261-db4f8b0cac24/server/0.log"
Oct 30 17:30:50.969893 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:50.969861    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gbc6r_b467741e-e532-40ad-9261-db4f8b0cac24/server/0.log"
Oct 30 17:30:51.729637 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:51.729606    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c9866f47-qfc2n_7966f66d-bd9f-48d7-905c-10ea4bd63c0c/console/0.log"
Oct 30 17:30:51.869056 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:51.869014    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-7h5xd_21066a6d-880f-4d1d-ad2b-9c143cd219af/download-server/0.log"
Oct 30 17:30:51.916420 ip-10-0-136-74 kubenswrapper[4497]: E1030 17:30:51.916384    4497 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6scbv" containerName="extractor"
Oct 30 17:30:52.842312 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:52.842283    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-bz58x_941eec69-d000-41b3-9250-3ce425d3f9e7/kube-auth-proxy/0.log"
Oct 30 17:30:52.996174 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:52.996138    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-jzt7r_c92b23e0-1211-499e-a41d-ab0ae8387659/istio-proxy/0.log"
Oct 30 17:30:55.864548 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:55.864514    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ts8b4_cfe1d12e-6cb7-487f-a352-590c8fcf7775/dns/0.log"
Oct 30 17:30:55.981795 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:55.981768    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ts8b4_cfe1d12e-6cb7-487f-a352-590c8fcf7775/kube-rbac-proxy/0.log"
Oct 30 17:30:56.104222 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:56.104166    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2lxwn_3a046e81-b1ab-4da2-ac50-7d06f2c30668/dns-node-resolver/0.log"
Oct 30 17:30:56.533471 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:56.533442    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-bz58x_941eec69-d000-41b3-9250-3ce425d3f9e7/kube-auth-proxy/0.log"
Oct 30 17:30:56.658382 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:56.658353    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-jzt7r_c92b23e0-1211-499e-a41d-ab0ae8387659/istio-proxy/0.log"
Oct 30 17:30:56.715203 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:56.715167    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-bz58x_941eec69-d000-41b3-9250-3ce425d3f9e7/kube-auth-proxy/0.log"
Oct 30 17:30:56.839223 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:56.839137    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-jzt7r_c92b23e0-1211-499e-a41d-ab0ae8387659/istio-proxy/0.log"
Oct 30 17:30:57.706611 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:57.706581    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-bz58x_941eec69-d000-41b3-9250-3ce425d3f9e7/kube-auth-proxy/0.log"
Oct 30 17:30:57.840976 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:57.840944    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-jzt7r_c92b23e0-1211-499e-a41d-ab0ae8387659/istio-proxy/0.log"
Oct 30 17:30:57.970935 ip-10-0-136-74 kubenswrapper[4497]: I1030 17:30:57.970861    4497 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rc86k_57050003-649e-4fad-ba51-9f1a534a859c/node-ca/0.log"